2023-02-19 12:58:16,389 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d 2023-02-19 12:58:16,402 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-19 12:58:16,418 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-19 12:58:16,419 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5, deleteOnExit=true 2023-02-19 12:58:16,419 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-19 12:58:16,419 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/test.cache.data in system properties and HBase conf 2023-02-19 12:58:16,420 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/hadoop.tmp.dir in system properties and HBase conf 2023-02-19 12:58:16,420 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/hadoop.log.dir in system properties and HBase conf 2023-02-19 12:58:16,421 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-19 12:58:16,421 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-19 12:58:16,422 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-19 12:58:16,541 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-19 12:58:16,907 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-19 12:58:16,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-19 12:58:16,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-19 12:58:16,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-19 12:58:16,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-19 12:58:16,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-19 12:58:16,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-19 12:58:16,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-19 12:58:16,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-19 12:58:16,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-19 12:58:16,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/nfs.dump.dir in system properties and HBase conf 2023-02-19 12:58:16,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/java.io.tmpdir in system properties and HBase conf 2023-02-19 12:58:16,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-19 12:58:16,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-19 12:58:16,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-19 12:58:17,423 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-19 12:58:17,427 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-19 12:58:17,705 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-19 12:58:17,854 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-19 12:58:17,869 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-19 12:58:17,904 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-19 12:58:17,935 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/java.io.tmpdir/Jetty_localhost_46341_hdfs____.3nhyjs/webapp 2023-02-19 12:58:18,084 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46341 2023-02-19 12:58:18,095 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-19 12:58:18,095 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-19 12:58:18,475 WARN [Listener at localhost/36945] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-19 12:58:18,576 WARN [Listener at localhost/36945] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-19 12:58:18,594 WARN [Listener at localhost/36945] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-19 12:58:18,601 INFO [Listener at localhost/36945] log.Slf4jLog(67): jetty-6.1.26 2023-02-19 12:58:18,605 INFO [Listener at localhost/36945] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/java.io.tmpdir/Jetty_localhost_44557_datanode____s97blq/webapp 2023-02-19 12:58:18,701 INFO [Listener at localhost/36945] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44557 2023-02-19 12:58:18,999 WARN [Listener at localhost/39017] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-19 12:58:19,013 WARN [Listener at localhost/39017] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-19 12:58:19,016 WARN [Listener at localhost/39017] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-19 12:58:19,018 INFO [Listener at localhost/39017] log.Slf4jLog(67): jetty-6.1.26 2023-02-19 12:58:19,023 INFO [Listener at localhost/39017] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/java.io.tmpdir/Jetty_localhost_34103_datanode____.wn1412/webapp 2023-02-19 12:58:19,119 INFO [Listener at localhost/39017] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34103 2023-02-19 12:58:19,128 WARN [Listener at localhost/45763] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-19 12:58:19,148 WARN [Listener at localhost/45763] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-19 12:58:19,152 WARN [Listener at localhost/45763] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-19 12:58:19,154 INFO [Listener at localhost/45763] log.Slf4jLog(67): jetty-6.1.26 2023-02-19 12:58:19,160 INFO [Listener at localhost/45763] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/java.io.tmpdir/Jetty_localhost_38423_datanode____.8c6qll/webapp 2023-02-19 12:58:19,277 INFO [Listener at localhost/45763] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38423 2023-02-19 12:58:19,286 WARN [Listener at localhost/43399] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-19 12:58:19,461 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xfb668efe8b6e1410: Processing first storage report for DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada from datanode 78813b76-a43f-4947-becb-21ca672de048 2023-02-19 12:58:19,463 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xfb668efe8b6e1410: from storage DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada node DatanodeRegistration(127.0.0.1:43099, datanodeUuid=78813b76-a43f-4947-becb-21ca672de048, infoPort=33113, infoSecurePort=0, ipcPort=43399, storageInfo=lv=-57;cid=testClusterID;nsid=628248349;c=1676811497500), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-19 12:58:19,463 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd4cba2de0d49057: Processing first storage report for DS-d33d234a-297e-4a34-8409-ee4b3b23a654 from datanode 71e52b31-5255-4bd9-9704-0a46b7647275 2023-02-19 12:58:19,463 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd4cba2de0d49057: from storage DS-d33d234a-297e-4a34-8409-ee4b3b23a654 node DatanodeRegistration(127.0.0.1:37433, datanodeUuid=71e52b31-5255-4bd9-9704-0a46b7647275, infoPort=44195, infoSecurePort=0, ipcPort=39017, storageInfo=lv=-57;cid=testClusterID;nsid=628248349;c=1676811497500), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-19 12:58:19,463 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb032e1b9876ec0af: Processing first storage report for DS-d5a8f8a2-1e96-4e4c-9714-153f20440514 from datanode f708cb9d-f08b-4be3-ba60-a65026033f70 2023-02-19 12:58:19,463 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb032e1b9876ec0af: from storage DS-d5a8f8a2-1e96-4e4c-9714-153f20440514 node DatanodeRegistration(127.0.0.1:36009, datanodeUuid=f708cb9d-f08b-4be3-ba60-a65026033f70, infoPort=35705, infoSecurePort=0, ipcPort=45763, storageInfo=lv=-57;cid=testClusterID;nsid=628248349;c=1676811497500), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-19 12:58:19,464 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xfb668efe8b6e1410: Processing first storage report for DS-6af51a7b-ce4d-45cf-a9ea-dda4cf2e3874 from datanode 78813b76-a43f-4947-becb-21ca672de048 2023-02-19 12:58:19,464 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xfb668efe8b6e1410: from storage DS-6af51a7b-ce4d-45cf-a9ea-dda4cf2e3874 node DatanodeRegistration(127.0.0.1:43099, datanodeUuid=78813b76-a43f-4947-becb-21ca672de048, infoPort=33113, infoSecurePort=0, ipcPort=43399, storageInfo=lv=-57;cid=testClusterID;nsid=628248349;c=1676811497500), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-19 12:58:19,464 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd4cba2de0d49057: Processing first storage report for DS-9ad754cc-1466-493d-b819-e7c0b24fa08a from datanode 71e52b31-5255-4bd9-9704-0a46b7647275 2023-02-19 12:58:19,464 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd4cba2de0d49057: from storage DS-9ad754cc-1466-493d-b819-e7c0b24fa08a node DatanodeRegistration(127.0.0.1:37433, datanodeUuid=71e52b31-5255-4bd9-9704-0a46b7647275, infoPort=44195, infoSecurePort=0, ipcPort=39017, storageInfo=lv=-57;cid=testClusterID;nsid=628248349;c=1676811497500), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-19 12:58:19,464 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb032e1b9876ec0af: Processing first storage report for DS-79626e51-24f3-4fd1-b660-13be4ea3f131 from datanode f708cb9d-f08b-4be3-ba60-a65026033f70 2023-02-19 12:58:19,464 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb032e1b9876ec0af: from storage DS-79626e51-24f3-4fd1-b660-13be4ea3f131 node DatanodeRegistration(127.0.0.1:36009, datanodeUuid=f708cb9d-f08b-4be3-ba60-a65026033f70, infoPort=35705, infoSecurePort=0, ipcPort=45763, storageInfo=lv=-57;cid=testClusterID;nsid=628248349;c=1676811497500), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-19 12:58:19,664 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d 2023-02-19 12:58:19,723 INFO [Listener at localhost/43399] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/zookeeper_0, clientPort=60853, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-19 12:58:19,734 INFO [Listener at localhost/43399] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60853 2023-02-19 12:58:19,745 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:19,748 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:20,405 INFO [Listener at localhost/43399] util.FSUtils(479): Created version file at hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461 with version=8 2023-02-19 12:58:20,406 INFO [Listener at localhost/43399] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/hbase-staging 2023-02-19 12:58:20,702 INFO [Listener at localhost/43399] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-19 12:58:21,160 INFO [Listener at localhost/43399] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-19 12:58:21,190 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,191 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,191 INFO [Listener at localhost/43399] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-19 12:58:21,192 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,192 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-19 12:58:21,332 INFO [Listener at localhost/43399] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-19 12:58:21,410 DEBUG [Listener at localhost/43399] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-19 12:58:21,507 INFO [Listener at localhost/43399] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34259 2023-02-19 12:58:21,517 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,520 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,541 INFO [Listener at localhost/43399] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34259 connecting to ZooKeeper ensemble=127.0.0.1:60853 2023-02-19 12:58:21,581 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:342590x0, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-19 12:58:21,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34259-0x1006ca8e2820000 connected 2023-02-19 12:58:21,606 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-19 12:58:21,607 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:21,610 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-19 12:58:21,618 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34259 2023-02-19 12:58:21,619 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34259 2023-02-19 12:58:21,619 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34259 2023-02-19 12:58:21,620 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34259 2023-02-19 12:58:21,620 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34259 2023-02-19 12:58:21,626 INFO [Listener at localhost/43399] master.HMaster(444): hbase.rootdir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461, hbase.cluster.distributed=false 2023-02-19 12:58:21,697 INFO [Listener at localhost/43399] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-19 12:58:21,697 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,697 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,697 INFO [Listener at localhost/43399] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-19 12:58:21,697 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,697 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-19 12:58:21,702 INFO [Listener at localhost/43399] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-19 12:58:21,705 INFO [Listener at localhost/43399] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41307 2023-02-19 12:58:21,707 INFO [Listener at localhost/43399] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-19 12:58:21,712 INFO [Listener at localhost/43399] mob.MobFileCache(124): MobFileCache disabled 2023-02-19 12:58:21,713 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,715 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,716 INFO [Listener at localhost/43399] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41307 connecting to ZooKeeper ensemble=127.0.0.1:60853 2023-02-19 12:58:21,720 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:413070x0, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-19 12:58:21,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41307-0x1006ca8e2820001 connected 2023-02-19 12:58:21,721 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-19 12:58:21,722 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:21,723 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-19 12:58:21,724 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41307 2023-02-19 12:58:21,724 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41307 2023-02-19 12:58:21,725 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41307 2023-02-19 12:58:21,725 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41307 2023-02-19 12:58:21,725 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41307 2023-02-19 12:58:21,738 INFO [Listener at localhost/43399] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-19 12:58:21,738 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,738 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,739 INFO [Listener at localhost/43399] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-19 12:58:21,739 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,739 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-19 12:58:21,739 INFO [Listener at localhost/43399] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-19 12:58:21,740 INFO [Listener at localhost/43399] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39199 2023-02-19 12:58:21,741 INFO [Listener at localhost/43399] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-19 12:58:21,742 INFO [Listener at localhost/43399] mob.MobFileCache(124): MobFileCache disabled 2023-02-19 12:58:21,742 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,745 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,746 INFO [Listener at localhost/43399] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39199 connecting to ZooKeeper ensemble=127.0.0.1:60853 2023-02-19 12:58:21,751 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:391990x0, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-19 12:58:21,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39199-0x1006ca8e2820002 connected 2023-02-19 12:58:21,752 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-19 12:58:21,753 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:21,754 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-19 12:58:21,754 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39199 2023-02-19 12:58:21,755 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39199 2023-02-19 12:58:21,755 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39199 2023-02-19 12:58:21,756 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39199 2023-02-19 12:58:21,756 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39199 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-19 12:58:21,770 INFO [Listener at localhost/43399] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-19 12:58:21,772 INFO [Listener at localhost/43399] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45077 2023-02-19 12:58:21,773 INFO [Listener at localhost/43399] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-19 12:58:21,773 INFO [Listener at localhost/43399] mob.MobFileCache(124): MobFileCache disabled 2023-02-19 12:58:21,774 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,776 INFO [Listener at localhost/43399] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,777 INFO [Listener at localhost/43399] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45077 connecting to ZooKeeper ensemble=127.0.0.1:60853 2023-02-19 12:58:21,783 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:450770x0, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-19 12:58:21,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45077-0x1006ca8e2820003 connected 2023-02-19 12:58:21,784 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-19 12:58:21,784 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:21,785 DEBUG [Listener at localhost/43399] zookeeper.ZKUtil(164): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-19 12:58:21,786 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45077 2023-02-19 12:58:21,786 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45077 2023-02-19 12:58:21,786 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45077 2023-02-19 12:58:21,787 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45077 2023-02-19 12:58:21,787 DEBUG [Listener at localhost/43399] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45077 2023-02-19 12:58:21,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:21,797 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-19 12:58:21,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:21,817 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-19 12:58:21,817 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-19 12:58:21,817 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-19 12:58:21,817 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-19 12:58:21,817 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:21,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-19 12:58:21,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34259,1676811500546 from backup master directory 2023-02-19 12:58:21,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-19 12:58:21,824 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:21,824 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-19 12:58:21,824 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-19 12:58:21,824 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:21,827 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-19 12:58:21,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-19 12:58:21,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/hbase.id with ID: 6b4531ed-8a20-4404-8f60-3bdefe4c18e8 2023-02-19 12:58:21,958 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-19 12:58:21,974 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:22,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2961df43 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:22,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c99d0b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:22,073 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-19 12:58:22,075 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-19 12:58:22,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-19 12:58:22,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-19 12:58:22,094 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-19 12:58:22,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-19 12:58:22,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-19 12:58:22,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store-tmp 2023-02-19 12:58:22,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:22,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-19 12:58:22,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:22,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:22,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-19 12:58:22,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:22,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:22,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-19 12:58:22,167 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/WALs/jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:22,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34259%2C1676811500546, suffix=, logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/WALs/jenkins-hbase4.apache.org,34259,1676811500546, archiveDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/oldWALs, maxLogs=10 2023-02-19 12:58:22,239 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK] 2023-02-19 12:58:22,239 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK] 2023-02-19 12:58:22,239 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK] 2023-02-19 12:58:22,247 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-19 12:58:22,310 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/WALs/jenkins-hbase4.apache.org,34259,1676811500546/jenkins-hbase4.apache.org%2C34259%2C1676811500546.1676811502195 2023-02-19 12:58:22,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK], DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK], DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK]] 2023-02-19 12:58:22,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-19 12:58:22,312 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:22,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-19 12:58:22,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-19 12:58:22,372 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-19 12:58:22,379 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-19 12:58:22,403 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-19 12:58:22,415 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:22,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-19 12:58:22,422 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-19 12:58:22,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-19 12:58:22,440 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:22,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69143299, jitterRate=0.030315443873405457}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:22,441 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-19 12:58:22,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-19 12:58:22,460 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-19 12:58:22,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-19 12:58:22,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-19 12:58:22,464 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-19 12:58:22,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 30 msec 2023-02-19 12:58:22,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-19 12:58:22,520 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-19 12:58:22,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-19 12:58:22,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-19 12:58:22,560 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-19 12:58:22,562 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-19 12:58:22,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-19 12:58:22,572 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-19 12:58:22,574 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:22,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-19 12:58:22,577 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-19 12:58:22,588 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-19 12:58:22,592 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:22,592 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:22,592 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:22,592 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:22,592 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:22,595 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34259,1676811500546, sessionid=0x1006ca8e2820000, setting cluster-up flag (Was=false) 2023-02-19 12:58:22,609 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:22,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-19 12:58:22,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:22,621 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:22,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-19 12:58:22,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:22,629 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp 2023-02-19 12:58:22,690 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(952): ClusterId : 6b4531ed-8a20-4404-8f60-3bdefe4c18e8 2023-02-19 12:58:22,690 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(952): ClusterId : 6b4531ed-8a20-4404-8f60-3bdefe4c18e8 2023-02-19 12:58:22,690 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(952): ClusterId : 6b4531ed-8a20-4404-8f60-3bdefe4c18e8 2023-02-19 12:58:22,696 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-19 12:58:22,696 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-19 12:58:22,696 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-19 12:58:22,702 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-19 12:58:22,702 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-19 12:58:22,702 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-19 12:58:22,703 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-19 12:58:22,702 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-19 12:58:22,703 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-19 12:58:22,708 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-19 12:58:22,708 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-19 12:58:22,708 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-19 12:58:22,712 DEBUG [RS:0;jenkins-hbase4:41307] zookeeper.ReadOnlyZKClient(139): Connect 0x1cfee55f to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:22,713 DEBUG [RS:2;jenkins-hbase4:45077] zookeeper.ReadOnlyZKClient(139): Connect 0x32fd08ad to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:22,713 DEBUG [RS:1;jenkins-hbase4:39199] zookeeper.ReadOnlyZKClient(139): Connect 0x280df3d3 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:22,745 DEBUG [RS:0;jenkins-hbase4:41307] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@123918b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:22,746 DEBUG [RS:0;jenkins-hbase4:41307] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16068e2f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-19 12:58:22,746 DEBUG [RS:2;jenkins-hbase4:45077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c3af082, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:22,746 DEBUG [RS:2;jenkins-hbase4:45077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@380563dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-19 12:58:22,749 DEBUG [RS:1;jenkins-hbase4:39199] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26ca08a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:22,749 DEBUG [RS:1;jenkins-hbase4:39199] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22f598a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-19 12:58:22,769 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39199 2023-02-19 12:58:22,769 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:45077 2023-02-19 12:58:22,772 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-19 12:58:22,773 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41307 2023-02-19 12:58:22,773 INFO [RS:2;jenkins-hbase4:45077] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-19 12:58:22,773 INFO [RS:1;jenkins-hbase4:39199] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-19 12:58:22,773 INFO [RS:1;jenkins-hbase4:39199] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-19 12:58:22,773 INFO [RS:0;jenkins-hbase4:41307] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-19 12:58:22,773 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1023): About to register with Master. 2023-02-19 12:58:22,773 INFO [RS:2;jenkins-hbase4:45077] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-19 12:58:22,774 INFO [RS:0;jenkins-hbase4:41307] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-19 12:58:22,774 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1023): About to register with Master. 2023-02-19 12:58:22,774 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1023): About to register with Master. 2023-02-19 12:58:22,776 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34259,1676811500546 with isa=jenkins-hbase4.apache.org/172.31.14.131:45077, startcode=1676811501769 2023-02-19 12:58:22,776 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34259,1676811500546 with isa=jenkins-hbase4.apache.org/172.31.14.131:39199, startcode=1676811501737 2023-02-19 12:58:22,776 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34259,1676811500546 with isa=jenkins-hbase4.apache.org/172.31.14.131:41307, startcode=1676811501696 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-19 12:58:22,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:22,787 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676811532786 2023-02-19 12:58:22,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-19 12:58:22,791 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-19 12:58:22,791 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-19 12:58:22,796 DEBUG [RS:0;jenkins-hbase4:41307] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-19 12:58:22,796 DEBUG [RS:1;jenkins-hbase4:39199] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-19 12:58:22,796 DEBUG [RS:2;jenkins-hbase4:45077] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-19 12:58:22,797 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-19 12:58:22,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-19 12:58:22,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-19 12:58:22,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-19 12:58:22,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-19 12:58:22,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-19 12:58:22,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-19 12:58:22,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-19 12:58:22,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-19 12:58:22,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-19 12:58:22,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-19 12:58:22,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676811502832,5,FailOnTimeoutGroup] 2023-02-19 12:58:22,833 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676811502832,5,FailOnTimeoutGroup] 2023-02-19 12:58:22,833 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,833 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-19 12:58:22,835 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,835 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,870 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:22,872 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:22,872 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461 2023-02-19 12:58:22,879 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60073, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-19 12:58:22,879 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42245, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-19 12:58:22,879 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57895, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-19 12:58:22,899 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:22,901 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-19 12:58:22,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34259] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:22,902 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:22,903 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34259] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:22,904 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/info 2023-02-19 12:58:22,905 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-19 12:58:22,906 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:22,906 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-19 12:58:22,908 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/rep_barrier 2023-02-19 12:58:22,909 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-19 12:58:22,910 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:22,910 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-19 12:58:22,912 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/table 2023-02-19 12:58:22,913 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-19 12:58:22,915 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:22,917 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740 2023-02-19 12:58:22,918 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740 2023-02-19 12:58:22,920 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461 2023-02-19 12:58:22,920 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461 2023-02-19 12:58:22,920 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461 2023-02-19 12:58:22,920 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36945 2023-02-19 12:58:22,920 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36945 2023-02-19 12:58:22,920 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-19 12:58:22,920 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36945 2023-02-19 12:58:22,920 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-19 12:58:22,920 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-19 12:58:22,923 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-19 12:58:22,925 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-19 12:58:22,926 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:22,930 DEBUG [RS:2;jenkins-hbase4:45077] zookeeper.ZKUtil(162): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:22,931 WARN [RS:2;jenkins-hbase4:45077] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-19 12:58:22,931 INFO [RS:2;jenkins-hbase4:45077] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-19 12:58:22,931 DEBUG [RS:1;jenkins-hbase4:39199] zookeeper.ZKUtil(162): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:22,931 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:22,931 DEBUG [RS:0;jenkins-hbase4:41307] zookeeper.ZKUtil(162): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:22,931 WARN [RS:1;jenkins-hbase4:39199] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-19 12:58:22,931 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:22,931 INFO [RS:1;jenkins-hbase4:39199] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-19 12:58:22,931 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41307,1676811501696] 2023-02-19 12:58:22,932 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:22,931 WARN [RS:0;jenkins-hbase4:41307] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-19 12:58:22,932 INFO [RS:0;jenkins-hbase4:41307] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-19 12:58:22,933 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:22,932 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45077,1676811501769] 2023-02-19 12:58:22,933 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71614638, jitterRate=0.06714126467704773}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-19 12:58:22,934 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39199,1676811501737] 2023-02-19 12:58:22,934 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-19 12:58:22,934 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-19 12:58:22,935 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-19 12:58:22,937 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-19 12:58:22,937 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-19 12:58:22,937 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-19 12:58:22,939 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-19 12:58:22,939 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-19 12:58:22,947 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-19 12:58:22,947 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-19 12:58:22,949 DEBUG [RS:0;jenkins-hbase4:41307] zookeeper.ZKUtil(162): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:22,949 DEBUG [RS:2;jenkins-hbase4:45077] zookeeper.ZKUtil(162): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:22,949 DEBUG [RS:1;jenkins-hbase4:39199] zookeeper.ZKUtil(162): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:22,950 DEBUG [RS:0;jenkins-hbase4:41307] zookeeper.ZKUtil(162): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:22,950 DEBUG [RS:1;jenkins-hbase4:39199] zookeeper.ZKUtil(162): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:22,950 DEBUG [RS:2;jenkins-hbase4:45077] zookeeper.ZKUtil(162): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:22,951 DEBUG [RS:0;jenkins-hbase4:41307] zookeeper.ZKUtil(162): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:22,951 DEBUG [RS:2;jenkins-hbase4:45077] zookeeper.ZKUtil(162): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:22,956 DEBUG [RS:1;jenkins-hbase4:39199] zookeeper.ZKUtil(162): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:22,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-19 12:58:22,962 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-19 12:58:22,962 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-19 12:58:22,962 DEBUG [RS:0;jenkins-hbase4:41307] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-19 12:58:22,971 INFO [RS:0;jenkins-hbase4:41307] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-19 12:58:22,972 INFO [RS:1;jenkins-hbase4:39199] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-19 12:58:22,972 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-19 12:58:22,971 INFO [RS:2;jenkins-hbase4:45077] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-19 12:58:22,975 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:22,993 INFO [RS:0;jenkins-hbase4:41307] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-19 12:58:22,993 INFO [RS:1;jenkins-hbase4:39199] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-19 12:58:22,993 INFO [RS:2;jenkins-hbase4:45077] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-19 12:58:22,996 INFO [RS:2;jenkins-hbase4:45077] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-19 12:58:22,996 INFO [RS:1;jenkins-hbase4:39199] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-19 12:58:22,997 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,996 INFO [RS:0;jenkins-hbase4:41307] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-19 12:58:22,997 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,997 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:22,998 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-19 12:58:22,998 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-19 12:58:22,998 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-19 12:58:23,006 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,006 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,006 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,006 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,006 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-19 12:58:23,007 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,007 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:2;jenkins-hbase4:45077] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,008 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-19 12:58:23,009 DEBUG [RS:1;jenkins-hbase4:39199] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,009 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,009 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,009 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,009 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,009 DEBUG [RS:0;jenkins-hbase4:41307] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-19 12:58:23,009 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,010 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,010 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,010 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,010 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,011 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,011 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,011 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,030 INFO [RS:0;jenkins-hbase4:41307] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-19 12:58:23,030 INFO [RS:1;jenkins-hbase4:39199] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-19 12:58:23,030 INFO [RS:2;jenkins-hbase4:45077] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-19 12:58:23,033 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45077,1676811501769-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,033 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41307,1676811501696-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,033 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39199,1676811501737-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,054 INFO [RS:0;jenkins-hbase4:41307] regionserver.Replication(203): jenkins-hbase4.apache.org,41307,1676811501696 started 2023-02-19 12:58:23,054 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41307,1676811501696, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41307, sessionid=0x1006ca8e2820001 2023-02-19 12:58:23,054 INFO [RS:1;jenkins-hbase4:39199] regionserver.Replication(203): jenkins-hbase4.apache.org,39199,1676811501737 started 2023-02-19 12:58:23,054 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39199,1676811501737, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39199, sessionid=0x1006ca8e2820002 2023-02-19 12:58:23,054 INFO [RS:2;jenkins-hbase4:45077] regionserver.Replication(203): jenkins-hbase4.apache.org,45077,1676811501769 started 2023-02-19 12:58:23,054 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-19 12:58:23,054 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-19 12:58:23,054 DEBUG [RS:0;jenkins-hbase4:41307] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:23,054 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,45077,1676811501769, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45077, sessionid=0x1006ca8e2820003 2023-02-19 12:58:23,055 DEBUG [RS:0;jenkins-hbase4:41307] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41307,1676811501696' 2023-02-19 12:58:23,055 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-19 12:58:23,054 DEBUG [RS:1;jenkins-hbase4:39199] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:23,055 DEBUG [RS:2;jenkins-hbase4:45077] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:23,055 DEBUG [RS:2;jenkins-hbase4:45077] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45077,1676811501769' 2023-02-19 12:58:23,055 DEBUG [RS:2;jenkins-hbase4:45077] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-19 12:58:23,055 DEBUG [RS:0;jenkins-hbase4:41307] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-19 12:58:23,055 DEBUG [RS:1;jenkins-hbase4:39199] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39199,1676811501737' 2023-02-19 12:58:23,056 DEBUG [RS:1;jenkins-hbase4:39199] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-19 12:58:23,056 DEBUG [RS:2;jenkins-hbase4:45077] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-19 12:58:23,056 DEBUG [RS:1;jenkins-hbase4:39199] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-19 12:58:23,056 DEBUG [RS:0;jenkins-hbase4:41307] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-19 12:58:23,057 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-19 12:58:23,057 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-19 12:58:23,057 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-19 12:58:23,057 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-19 12:58:23,057 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-19 12:58:23,057 DEBUG [RS:0;jenkins-hbase4:41307] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:23,057 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-19 12:58:23,058 DEBUG [RS:0;jenkins-hbase4:41307] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41307,1676811501696' 2023-02-19 12:58:23,058 DEBUG [RS:0;jenkins-hbase4:41307] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-19 12:58:23,057 DEBUG [RS:2;jenkins-hbase4:45077] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:23,058 DEBUG [RS:1;jenkins-hbase4:39199] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:23,058 DEBUG [RS:2;jenkins-hbase4:45077] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45077,1676811501769' 2023-02-19 12:58:23,058 DEBUG [RS:2;jenkins-hbase4:45077] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-19 12:58:23,058 DEBUG [RS:1;jenkins-hbase4:39199] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39199,1676811501737' 2023-02-19 12:58:23,058 DEBUG [RS:1;jenkins-hbase4:39199] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-19 12:58:23,058 DEBUG [RS:0;jenkins-hbase4:41307] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-19 12:58:23,059 DEBUG [RS:2;jenkins-hbase4:45077] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-19 12:58:23,059 DEBUG [RS:1;jenkins-hbase4:39199] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-19 12:58:23,059 DEBUG [RS:0;jenkins-hbase4:41307] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-19 12:58:23,059 INFO [RS:0;jenkins-hbase4:41307] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-19 12:58:23,059 INFO [RS:0;jenkins-hbase4:41307] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-19 12:58:23,059 DEBUG [RS:2;jenkins-hbase4:45077] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-19 12:58:23,059 DEBUG [RS:1;jenkins-hbase4:39199] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-19 12:58:23,059 INFO [RS:2;jenkins-hbase4:45077] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-19 12:58:23,060 INFO [RS:1;jenkins-hbase4:39199] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-19 12:58:23,060 INFO [RS:1;jenkins-hbase4:39199] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-19 12:58:23,060 INFO [RS:2;jenkins-hbase4:45077] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-19 12:58:23,127 DEBUG [jenkins-hbase4:34259] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-19 12:58:23,131 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:23,137 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:23,137 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:23,137 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:23,137 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:23,139 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45077,1676811501769, state=OPENING 2023-02-19 12:58:23,147 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-19 12:58:23,148 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:23,149 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-19 12:58:23,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:23,172 INFO [RS:0;jenkins-hbase4:41307] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41307%2C1676811501696, suffix=, logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,41307,1676811501696, archiveDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs, maxLogs=32 2023-02-19 12:58:23,172 INFO [RS:1;jenkins-hbase4:39199] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39199%2C1676811501737, suffix=, logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,39199,1676811501737, archiveDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs, maxLogs=32 2023-02-19 12:58:23,174 INFO [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45077%2C1676811501769, suffix=, logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,45077,1676811501769, archiveDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs, maxLogs=32 2023-02-19 12:58:23,203 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK] 2023-02-19 12:58:23,203 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK] 2023-02-19 12:58:23,204 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK] 2023-02-19 12:58:23,206 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK] 2023-02-19 12:58:23,207 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK] 2023-02-19 12:58:23,207 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK] 2023-02-19 12:58:23,214 INFO [RS:0;jenkins-hbase4:41307] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,41307,1676811501696/jenkins-hbase4.apache.org%2C41307%2C1676811501696.1676811503177 2023-02-19 12:58:23,214 INFO [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,45077,1676811501769/jenkins-hbase4.apache.org%2C45077%2C1676811501769.1676811503177 2023-02-19 12:58:23,214 DEBUG [RS:0;jenkins-hbase4:41307] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK], DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK], DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK]] 2023-02-19 12:58:23,216 DEBUG [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK], DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK], DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK]] 2023-02-19 12:58:23,219 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK] 2023-02-19 12:58:23,219 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK] 2023-02-19 12:58:23,219 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK] 2023-02-19 12:58:23,225 INFO [RS:1;jenkins-hbase4:39199] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,39199,1676811501737/jenkins-hbase4.apache.org%2C39199%2C1676811501737.1676811503177 2023-02-19 12:58:23,226 DEBUG [RS:1;jenkins-hbase4:39199] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK], DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK], DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK]] 2023-02-19 12:58:23,337 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:23,340 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-19 12:58:23,343 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57838, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-19 12:58:23,358 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-19 12:58:23,358 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-19 12:58:23,361 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45077%2C1676811501769.meta, suffix=.meta, logDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,45077,1676811501769, archiveDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs, maxLogs=32 2023-02-19 12:58:23,382 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK] 2023-02-19 12:58:23,383 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK] 2023-02-19 12:58:23,384 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK] 2023-02-19 12:58:23,391 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/WALs/jenkins-hbase4.apache.org,45077,1676811501769/jenkins-hbase4.apache.org%2C45077%2C1676811501769.meta.1676811503363.meta 2023-02-19 12:58:23,392 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43099,DS-00b2ac0d-44a3-4371-aa83-ed6e80bd2ada,DISK], DatanodeInfoWithStorage[127.0.0.1:37433,DS-d33d234a-297e-4a34-8409-ee4b3b23a654,DISK], DatanodeInfoWithStorage[127.0.0.1:36009,DS-d5a8f8a2-1e96-4e4c-9714-153f20440514,DISK]] 2023-02-19 12:58:23,392 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-19 12:58:23,394 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-19 12:58:23,411 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-19 12:58:23,416 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-19 12:58:23,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-19 12:58:23,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:23,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-19 12:58:23,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-19 12:58:23,424 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-19 12:58:23,426 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/info 2023-02-19 12:58:23,426 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/info 2023-02-19 12:58:23,427 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-19 12:58:23,428 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:23,428 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-19 12:58:23,429 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/rep_barrier 2023-02-19 12:58:23,429 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/rep_barrier 2023-02-19 12:58:23,430 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-19 12:58:23,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:23,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-19 12:58:23,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/table 2023-02-19 12:58:23,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/table 2023-02-19 12:58:23,433 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-19 12:58:23,433 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:23,436 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740 2023-02-19 12:58:23,438 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740 2023-02-19 12:58:23,442 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-19 12:58:23,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-19 12:58:23,446 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68411144, jitterRate=0.019405484199523926}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-19 12:58:23,447 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-19 12:58:23,458 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676811503330 2023-02-19 12:58:23,476 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-19 12:58:23,476 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-19 12:58:23,477 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45077,1676811501769, state=OPEN 2023-02-19 12:58:23,479 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-19 12:58:23,479 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-19 12:58:23,484 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-19 12:58:23,484 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45077,1676811501769 in 327 msec 2023-02-19 12:58:23,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-19 12:58:23,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 527 msec 2023-02-19 12:58:23,494 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 821 msec 2023-02-19 12:58:23,494 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1676811503494, completionTime=-1 2023-02-19 12:58:23,494 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-19 12:58:23,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-19 12:58:23,559 DEBUG [hconnection-0x514281c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:23,562 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50358, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:23,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-19 12:58:23,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676811563580 2023-02-19 12:58:23,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676811623580 2023-02-19 12:58:23,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 86 msec 2023-02-19 12:58:23,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34259,1676811500546-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34259,1676811500546-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34259,1676811500546-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34259, period=300000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,607 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-19 12:58:23,616 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-19 12:58:23,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-19 12:58:23,626 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-19 12:58:23,636 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-19 12:58:23,638 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-19 12:58:23,641 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-19 12:58:23,663 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:23,666 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1 empty. 2023-02-19 12:58:23,667 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:23,667 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-19 12:58:23,706 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:23,708 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 27c253a2f5b636a8349cb7b2efa921a1, NAME => 'hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:23,727 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:23,727 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 27c253a2f5b636a8349cb7b2efa921a1, disabling compactions & flushes 2023-02-19 12:58:23,727 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:23,727 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:23,727 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. after waiting 0 ms 2023-02-19 12:58:23,727 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:23,727 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:23,727 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 27c253a2f5b636a8349cb7b2efa921a1: 2023-02-19 12:58:23,732 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-19 12:58:23,747 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676811503735"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811503735"}]},"ts":"1676811503735"} 2023-02-19 12:58:23,777 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-19 12:58:23,779 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-19 12:58:23,784 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811503780"}]},"ts":"1676811503780"} 2023-02-19 12:58:23,788 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-19 12:58:23,792 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:23,793 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:23,793 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:23,793 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:23,793 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:23,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=27c253a2f5b636a8349cb7b2efa921a1, ASSIGN}] 2023-02-19 12:58:23,799 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=27c253a2f5b636a8349cb7b2efa921a1, ASSIGN 2023-02-19 12:58:23,800 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=27c253a2f5b636a8349cb7b2efa921a1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:23,953 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-19 12:58:23,954 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=27c253a2f5b636a8349cb7b2efa921a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:23,954 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676811503954"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811503954"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811503954"}]},"ts":"1676811503954"} 2023-02-19 12:58:23,959 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 27c253a2f5b636a8349cb7b2efa921a1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:24,114 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:24,114 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-19 12:58:24,119 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51986, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-19 12:58:24,125 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:24,126 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 27c253a2f5b636a8349cb7b2efa921a1, NAME => 'hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.', STARTKEY => '', ENDKEY => ''} 2023-02-19 12:58:24,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,129 INFO [StoreOpener-27c253a2f5b636a8349cb7b2efa921a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,131 DEBUG [StoreOpener-27c253a2f5b636a8349cb7b2efa921a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/info 2023-02-19 12:58:24,131 DEBUG [StoreOpener-27c253a2f5b636a8349cb7b2efa921a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/info 2023-02-19 12:58:24,132 INFO [StoreOpener-27c253a2f5b636a8349cb7b2efa921a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 27c253a2f5b636a8349cb7b2efa921a1 columnFamilyName info 2023-02-19 12:58:24,132 INFO [StoreOpener-27c253a2f5b636a8349cb7b2efa921a1-1] regionserver.HStore(310): Store=27c253a2f5b636a8349cb7b2efa921a1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:24,134 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,135 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:24,143 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:24,144 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 27c253a2f5b636a8349cb7b2efa921a1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73782171, jitterRate=0.09944002330303192}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:24,144 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 27c253a2f5b636a8349cb7b2efa921a1: 2023-02-19 12:58:24,146 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1., pid=6, masterSystemTime=1676811504114 2023-02-19 12:58:24,151 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:24,151 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:24,153 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=27c253a2f5b636a8349cb7b2efa921a1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:24,153 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676811504152"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811504152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811504152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811504152"}]},"ts":"1676811504152"} 2023-02-19 12:58:24,160 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-19 12:58:24,160 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 27c253a2f5b636a8349cb7b2efa921a1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 198 msec 2023-02-19 12:58:24,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-19 12:58:24,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=27c253a2f5b636a8349cb7b2efa921a1, ASSIGN in 364 msec 2023-02-19 12:58:24,166 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-19 12:58:24,166 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811504166"}]},"ts":"1676811504166"} 2023-02-19 12:58:24,168 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-19 12:58:24,173 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-19 12:58:24,176 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 546 msec 2023-02-19 12:58:24,239 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-19 12:58:24,240 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-19 12:58:24,240 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:24,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:24,265 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51998, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:24,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-19 12:58:24,298 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-19 12:58:24,305 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-02-19 12:58:24,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-19 12:58:24,326 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-19 12:58:24,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-19 12:58:24,339 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-19 12:58:24,343 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-19 12:58:24,343 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.519sec 2023-02-19 12:58:24,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-19 12:58:24,347 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-19 12:58:24,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-19 12:58:24,349 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34259,1676811500546-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-19 12:58:24,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34259,1676811500546-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-19 12:58:24,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-02-19 12:58:24,397 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(139): Connect 0x297db6e1 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:24,401 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c633018, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:24,416 DEBUG [hconnection-0x6de71207-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:24,427 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50368, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:24,437 INFO [Listener at localhost/43399] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:24,465 INFO [Listener at localhost/43399] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=359, OpenFileDescriptor=695, MaxFileDescriptor=60000, SystemLoadAverage=137, ProcessCount=169, AvailableMemoryMB=9877 2023-02-19 12:58:24,479 DEBUG [Listener at localhost/43399] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-19 12:58:24,482 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-19 12:58:24,495 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-19 12:58:24,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:24,500 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676811504468 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-19 12:58:24,502 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676811504468 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-19 12:58:24,504 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676811504468" procId is: 9 2023-02-19 12:58:24,538 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:24,538 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:24,538 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:24,540 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:24,540 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:24,542 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:24,542 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f empty. 2023-02-19 12:58:24,542 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 empty. 2023-02-19 12:58:24,545 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 empty. 2023-02-19 12:58:24,547 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb empty. 2023-02-19 12:58:24,547 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:24,547 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 empty. 2023-02-19 12:58:24,548 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:24,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-19 12:58:24,548 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 empty. 2023-02-19 12:58:24,548 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:24,551 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:24,551 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:24,551 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:24,551 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676811504468 regions 2023-02-19 12:58:24,572 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:24,575 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(7671): creating {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:24,575 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(7671): creating {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:24,575 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(7671): creating {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:24,613 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:24,614 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:24,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:24,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:24,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:24,614 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:24,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,614 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:24,615 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:24,616 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:24,616 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,616 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:24,616 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:24,616 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:24,616 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:24,616 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:24,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:24,616 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:24,617 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:24,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:24,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:24,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:24,617 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:24,617 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:24,618 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(7671): creating {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:24,618 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(7671): creating {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:24,646 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,647 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:24,647 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:24,647 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:24,647 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:24,647 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:24,647 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:24,647 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-2] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:24,648 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,649 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:24,649 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:24,649 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:24,649 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:24,649 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:24,649 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:24,649 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:24,649 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:24,650 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:24,650 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:24,650 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:24,650 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:24,650 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:24,650 INFO [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:24,650 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676811504468-pool-1] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:24,655 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676811504468 execute state=CREATE_TABLE_ADD_TO_META 2023-02-19 12:58:24,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811504656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811504656"}]},"ts":"1676811504656"} 2023-02-19 12:58:24,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811504656"}]},"ts":"1676811504656"} 2023-02-19 12:58:24,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811504656"}]},"ts":"1676811504656"} 2023-02-19 12:58:24,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811504656"}]},"ts":"1676811504656"} 2023-02-19 12:58:24,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811504656"}]},"ts":"1676811504656"} 2023-02-19 12:58:24,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811504656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811504656"}]},"ts":"1676811504656"} 2023-02-19 12:58:24,702 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-19 12:58:24,704 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676811504468 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-19 12:58:24,705 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811504704"}]},"ts":"1676811504704"} 2023-02-19 12:58:24,707 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:24,712 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:24,717 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:24,717 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:24,717 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:24,717 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:24,717 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:24,720 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:24,721 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:24,721 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:24,721 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:24,723 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:24,723 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=false 2023-02-19 12:58:24,723 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=false 2023-02-19 12:58:24,723 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:24,723 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=false 2023-02-19 12:58:24,724 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:24,725 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:24,726 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=false 2023-02-19 12:58:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-19 12:58:24,873 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:24,875 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:24,875 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:24,875 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:24,875 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:24,875 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:24,876 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811504875"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811504875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811504875"}]},"ts":"1676811504875"} 2023-02-19 12:58:24,876 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504875"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811504875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811504875"}]},"ts":"1676811504875"} 2023-02-19 12:58:24,876 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504875"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811504875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811504875"}]},"ts":"1676811504875"} 2023-02-19 12:58:24,876 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504875"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811504875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811504875"}]},"ts":"1676811504875"} 2023-02-19 12:58:24,876 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811504875"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811504875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811504875"}]},"ts":"1676811504875"} 2023-02-19 12:58:24,879 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=13, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:24,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:24,884 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=12, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:24,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=11, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:24,887 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:24,887 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811504887"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811504887"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811504887"}]},"ts":"1676811504887"} 2023-02-19 12:58:24,887 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=14, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:24,891 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=15, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:25,037 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:25,037 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-19 12:58:25,041 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36496, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-19 12:58:25,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:25,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:25,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,046 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:25,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:25,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:25,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:25,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,054 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:25,054 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:25,056 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,057 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,058 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:25,058 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:25,058 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:25,058 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:25,058 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:25,059 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:25,060 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:25,061 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:25,063 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:25,063 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:25,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:25,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:25,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:25,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66333821, jitterRate=-0.011549040675163269}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:25,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68357804, jitterRate=0.018610656261444092}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:25,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:25,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:25,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72168870, jitterRate=0.0753999650478363}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:25,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:25,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=21, masterSystemTime=1676811505039 2023-02-19 12:58:25,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=16, masterSystemTime=1676811505033 2023-02-19 12:58:25,089 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=20, masterSystemTime=1676811505037 2023-02-19 12:58:25,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:25,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:25,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,094 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:25,094 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811505094"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811505094"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811505094"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811505094"}]},"ts":"1676811505094"} 2023-02-19 12:58:25,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:25,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,096 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:25,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:25,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,096 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505095"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811505095"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811505095"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811505095"}]},"ts":"1676811505095"} 2023-02-19 12:58:25,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,098 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:25,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,099 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:25,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:25,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,099 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505099"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811505099"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811505099"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811505099"}]},"ts":"1676811505099"} 2023-02-19 12:58:25,102 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:25,102 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:25,103 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:25,104 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,105 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=15 2023-02-19 12:58:25,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=15, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 209 msec 2023-02-19 12:58:25,108 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:25,109 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:25,109 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:25,110 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:25,110 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:25,111 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:25,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=13 2023-02-19 12:58:25,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=13, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769 in 223 msec 2023-02-19 12:58:25,113 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:25,114 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 391 msec 2023-02-19 12:58:25,114 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:25,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,114 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=14 2023-02-19 12:58:25,115 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=14, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 220 msec 2023-02-19 12:58:25,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 396 msec 2023-02-19 12:58:25,120 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:25,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,122 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 401 msec 2023-02-19 12:58:25,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:25,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73459150, jitterRate=0.09462663531303406}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:25,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:25,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=19, masterSystemTime=1676811505039 2023-02-19 12:58:25,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,132 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:25,133 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505132"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811505132"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811505132"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811505132"}]},"ts":"1676811505132"} 2023-02-19 12:58:25,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:25,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:25,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62794032, jitterRate=-0.06429600715637207}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:25,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:25,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59376749, jitterRate=-0.11521749198436737}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:25,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:25,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=18, masterSystemTime=1676811505033 2023-02-19 12:58:25,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=17, masterSystemTime=1676811505037 2023-02-19 12:58:25,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,139 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=11 2023-02-19 12:58:25,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=11, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737 in 251 msec 2023-02-19 12:58:25,140 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:25,141 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505140"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811505140"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811505140"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811505140"}]},"ts":"1676811505140"} 2023-02-19 12:58:25,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,144 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:25,144 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811505143"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811505143"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811505143"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811505143"}]},"ts":"1676811505143"} 2023-02-19 12:58:25,145 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 423 msec 2023-02-19 12:58:25,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=12 2023-02-19 12:58:25,149 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=12, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 261 msec 2023-02-19 12:58:25,151 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-02-19 12:58:25,151 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 266 msec 2023-02-19 12:58:25,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 432 msec 2023-02-19 12:58:25,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-19 12:58:25,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 434 msec 2023-02-19 12:58:25,155 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676811504468 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-19 12:58:25,155 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811505155"}]},"ts":"1676811505155"} 2023-02-19 12:58:25,157 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:25,160 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676811504468 execute state=CREATE_TABLE_POST_OPERATION 2023-02-19 12:58:25,163 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676811504468 in 665 msec 2023-02-19 12:58:25,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-19 12:58:25,304 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 9 completed 2023-02-19 12:58:25,310 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676811504468 2023-02-19 12:58:25,311 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,314 INFO [Listener at localhost/43399] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-19 12:58:25,335 DEBUG [hconnection-0x6de71207-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:25,339 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:25,342 DEBUG [hconnection-0x6de71207-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:25,344 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52000, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:25,356 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:25,360 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:25,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:25,370 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811505370"}]},"ts":"1676811505370"} 2023-02-19 12:58:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-19 12:58:25,372 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:25,374 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:25,376 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:25,378 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:25,378 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:25,379 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:25,379 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:25,380 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:25,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:25,381 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:25,381 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:25,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811505381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811505381"}]},"ts":"1676811505381"} 2023-02-19 12:58:25,381 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811505381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811505381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811505381"}]},"ts":"1676811505381"} 2023-02-19 12:58:25,382 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811505381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811505381"}]},"ts":"1676811505381"} 2023-02-19 12:58:25,382 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:25,382 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505382"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811505382"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811505382"}]},"ts":"1676811505382"} 2023-02-19 12:58:25,383 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:25,383 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811505383"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811505383"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811505383"}]},"ts":"1676811505383"} 2023-02-19 12:58:25,384 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:25,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:25,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=25, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:25,387 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=26, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:25,390 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:25,397 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:25,398 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:25,398 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505398"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811505398"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811505398"}]},"ts":"1676811505398"} 2023-02-19 12:58:25,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:25,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:25,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:25,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:25,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:25,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:25,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:25,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:25,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:25,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:25,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:25,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:25,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:25,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,566 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:25,566 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505566"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811505566"}]},"ts":"1676811505566"} 2023-02-19 12:58:25,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:25,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,568 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:25,568 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505568"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811505568"}]},"ts":"1676811505568"} 2023-02-19 12:58:25,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:25,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:25,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:25,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,569 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:25,569 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811505569"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811505569"}]},"ts":"1676811505569"} 2023-02-19 12:58:25,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:25,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:25,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:25,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:25,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:25,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-19 12:58:25,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:25,579 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737 in 184 msec 2023-02-19 12:58:25,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:25,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=25 2023-02-19 12:58:25,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=25, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 184 msec 2023-02-19 12:58:25,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:25,581 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-19 12:58:25,581 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 189 msec 2023-02-19 12:58:25,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:25,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:25,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:25,582 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 203 msec 2023-02-19 12:58:25,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:25,583 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 203 msec 2023-02-19 12:58:25,583 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:25,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:25,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:25,583 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505583"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811505583"}]},"ts":"1676811505583"} 2023-02-19 12:58:25,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:25,584 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 205 msec 2023-02-19 12:58:25,585 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:25,585 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811505585"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811505585"}]},"ts":"1676811505585"} 2023-02-19 12:58:25,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:25,586 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:25,586 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811505586"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811505586"}]},"ts":"1676811505586"} 2023-02-19 12:58:25,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-19 12:58:25,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 185 msec 2023-02-19 12:58:25,591 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=26 2023-02-19 12:58:25,591 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769 in 200 msec 2023-02-19 12:58:25,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-19 12:58:25,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 198 msec 2023-02-19 12:58:25,592 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 214 msec 2023-02-19 12:58:25,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 215 msec 2023-02-19 12:58:25,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=22 2023-02-19 12:58:25,595 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 216 msec 2023-02-19 12:58:25,595 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811505595"}]},"ts":"1676811505595"} 2023-02-19 12:58:25,597 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:25,599 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:25,602 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 239 msec 2023-02-19 12:58:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-19 12:58:25,623 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 22 completed 2023-02-19 12:58:25,633 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811505633 (current time:1676811505633). 2023-02-19 12:58:25,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:25,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x220fe93d to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:25,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e2faa50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:25,647 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50374, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x220fe93d to 127.0.0.1:60853 2023-02-19 12:58:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:25,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:25,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:25,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:25,671 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:25,672 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE; timeout=600000, timestamp=1676812105672 2023-02-19 12:58:25,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:25,672 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676811504468 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 2023-02-19 12:58:25,677 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:25,677 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:25,694 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:25,700 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,701 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,701 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,701 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,701 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,704 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,704 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:25,707 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,708 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,708 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,707 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,707 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,707 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,707 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:25,715 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,715 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,715 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,715 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,716 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,716 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,716 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:25,927 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:25,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' is still in progress! 2023-02-19 12:58:25,932 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-19 12:58:26,188 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:26,189 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676811504468 2023-02-19 12:58:26,191 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811504468 2023-02-19 12:58:26,234 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811504468 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/emptySnaptb-1676811504468 2023-02-19 12:58:26,275 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676811504468 of table testRestoreSnapshot-1676811504468 completed 2023-02-19 12:58:26,275 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811504468 2023-02-19 12:58:26,275 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811504468 2023-02-19 12:58:26,276 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676811504468 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 at 1676811505672Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } at 1676811505694 (+22 ms)Consolidate snapshot: emptySnaptb-1676811504468 at 1676811506189 (+495 ms)Loading Region manifests for emptySnaptb-1676811504468 at 1676811506189Writing data manifest for emptySnaptb-1676811504468 at 1676811506203 (+14 ms)Verifying snapshot: emptySnaptb-1676811504468 at 1676811506221 (+18 ms)Snapshot emptySnaptb-1676811504468 of table testRestoreSnapshot-1676811504468 completed at 1676811506275 (+54 ms) 2023-02-19 12:58:26,277 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:26,279 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE in 614 msec 2023-02-19 12:58:26,432 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:26,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:26,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:26,436 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:26,440 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676811504468 2023-02-19 12:58:26,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:26,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-19 12:58:26,450 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676811504468 2023-02-19 12:58:26,450 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811506450"}]},"ts":"1676811506450"} 2023-02-19 12:58:26,452 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:26,456 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:26,458 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:26,460 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:26,460 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:26,460 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:26,461 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:26,462 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:26,462 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:26,463 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:26,463 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:26,463 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:26,463 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:26,463 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:26,463 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:26,464 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:26,464 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:26,465 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:26,465 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:26,466 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:26,466 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:26,612 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:26,613 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:26,613 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:26,613 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:26,613 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506612"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811506612"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811506612"}]},"ts":"1676811506612"} 2023-02-19 12:58:26,613 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811506613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811506613"}]},"ts":"1676811506613"} 2023-02-19 12:58:26,613 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506612"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811506612"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811506612"}]},"ts":"1676811506612"} 2023-02-19 12:58:26,613 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:26,613 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:26,613 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506612"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811506612"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811506612"}]},"ts":"1676811506612"} 2023-02-19 12:58:26,613 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811506613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811506613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811506613"}]},"ts":"1676811506613"} 2023-02-19 12:58:26,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=39, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:26,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=38, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:26,617 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=40, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:26,619 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=41, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:26,620 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:26,620 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811506620"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811506620"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811506620"}]},"ts":"1676811506620"} 2023-02-19 12:58:26,621 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=37, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:26,622 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=42, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:26,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-19 12:58:26,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:26,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:26,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:26,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:26,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:26,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:26,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,774 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,775 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:26,775 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:26,775 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,776 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:26,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:26,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:26,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:26,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,777 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:26,779 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:26,779 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:26,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,780 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,781 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:26,781 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:26,781 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:26,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,783 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:26,783 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:26,784 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:26,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:26,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70093427, jitterRate=0.04447345435619354}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:26,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:26,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=45, masterSystemTime=1676811506767 2023-02-19 12:58:26,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:26,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:26,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:26,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:26,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:26,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:26,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:26,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61621057, jitterRate=-0.08177469670772552}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:26,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:26,794 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:26,794 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506794"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811506794"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811506794"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811506794"}]},"ts":"1676811506794"} 2023-02-19 12:58:26,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=48, masterSystemTime=1676811506769 2023-02-19 12:58:26,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69156555, jitterRate=0.0305129736661911}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:26,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:26,798 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:26,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:26,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=47, masterSystemTime=1676811506772 2023-02-19 12:58:26,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:26,799 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:26,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:26,799 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:26,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:26,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,800 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:26,801 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:26,801 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811506801"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811506801"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811506801"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811506801"}]},"ts":"1676811506801"} 2023-02-19 12:58:26,802 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:26,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:26,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:26,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:26,810 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:26,810 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:26,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:26,810 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811506810"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811506810"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811506810"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811506810"}]},"ts":"1676811506810"} 2023-02-19 12:58:26,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,814 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:26,814 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:26,814 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=40 2023-02-19 12:58:26,814 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=40, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769 in 183 msec 2023-02-19 12:58:26,814 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:26,815 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,816 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:26,816 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:26,816 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:26,817 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:26,817 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 358 msec 2023-02-19 12:58:26,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=42 2023-02-19 12:58:26,819 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=42, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 192 msec 2023-02-19 12:58:26,819 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:26,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=37 2023-02-19 12:58:26,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=37, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 195 msec 2023-02-19 12:58:26,822 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 362 msec 2023-02-19 12:58:26,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:26,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,824 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 365 msec 2023-02-19 12:58:26,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71991555, jitterRate=0.0727577656507492}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:26,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:26,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:26,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=43, masterSystemTime=1676811506767 2023-02-19 12:58:26,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61345393, jitterRate=-0.08588241040706635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:26,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:26,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:26,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:26,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:26,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=44, masterSystemTime=1676811506769 2023-02-19 12:58:26,828 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:26,828 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506828"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811506828"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811506828"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811506828"}]},"ts":"1676811506828"} 2023-02-19 12:58:26,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66515667, jitterRate=-0.008839324116706848}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:26,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:26,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=46, masterSystemTime=1676811506772 2023-02-19 12:58:26,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:26,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:26,831 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:26,832 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506831"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811506831"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811506831"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811506831"}]},"ts":"1676811506831"} 2023-02-19 12:58:26,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:26,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:26,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=39 2023-02-19 12:58:26,835 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:26,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=39, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 216 msec 2023-02-19 12:58:26,835 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811506835"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811506835"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811506835"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811506835"}]},"ts":"1676811506835"} 2023-02-19 12:58:26,837 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 379 msec 2023-02-19 12:58:26,838 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=38 2023-02-19 12:58:26,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=38, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737 in 219 msec 2023-02-19 12:58:26,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 383 msec 2023-02-19 12:58:26,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=41 2023-02-19 12:58:26,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=41, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 219 msec 2023-02-19 12:58:26,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=36 2023-02-19 12:58:26,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 385 msec 2023-02-19 12:58:26,845 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811506845"}]},"ts":"1676811506845"} 2023-02-19 12:58:26,846 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:26,848 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676811504468' was successfully enabled. 2023-02-19 12:58:26,851 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676811504468 in 408 msec 2023-02-19 12:58:27,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-19 12:58:27,203 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 36 completed 2023-02-19 12:58:27,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:27,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:27,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:27,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:27,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:27,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:27,260 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676811504468 2023-02-19 12:58:27,260 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:27,262 INFO [Listener at localhost/43399] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-19 12:58:27,282 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:27,283 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:27,286 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:27,287 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:27,290 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:27,302 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:27,302 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:27,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:27,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-19 12:58:27,308 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811507308"}]},"ts":"1676811507308"} 2023-02-19 12:58:27,310 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:27,313 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:27,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:27,317 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:27,317 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:27,317 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:27,317 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:27,317 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:27,319 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:27,319 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:27,319 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811507319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811507319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811507319"}]},"ts":"1676811507319"} 2023-02-19 12:58:27,319 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:27,319 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811507319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811507319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811507319"}]},"ts":"1676811507319"} 2023-02-19 12:58:27,319 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811507319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811507319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811507319"}]},"ts":"1676811507319"} 2023-02-19 12:58:27,319 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:27,319 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:27,319 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811507319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811507319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811507319"}]},"ts":"1676811507319"} 2023-02-19 12:58:27,319 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811507319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811507319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811507319"}]},"ts":"1676811507319"} 2023-02-19 12:58:27,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:27,322 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=51, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:27,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=50, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:27,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=52, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:27,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=54, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:27,330 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:27,332 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:27,332 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811507332"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811507332"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811507332"}]},"ts":"1676811507332"} 2023-02-19 12:58:27,334 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=53, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:27,477 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:27,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:27,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:27,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:27,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:27,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:27,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:27,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:27,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:27,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:27,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:27,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:27,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f3edc0852f9dc085eb7e7bda3e53af28 1/1 column families, dataSize=22.63 KB heapSize=48.91 KB 2023-02-19 12:58:27,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 60f6a18c9ce621af4e18258a265d998f 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-19 12:58:27,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:27,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:27,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:27,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:27,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:27,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:27,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9073362bb9d4cd684678e7eb8e65b9f1 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-19 12:58:27,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-19 12:58:27,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:27,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:27,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:27,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/.tmp/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:27,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/.tmp/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:27,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/.tmp/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:27,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/.tmp/cf/60d7d801122244c2923fae4f69e56306 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:27,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/.tmp/cf/695ee31eaa9a460b80aa2b4fd5fca7dc as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:27,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/.tmp/cf/686fa623ebe140c593a3f885472e7cb2 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:27,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306, entries=346, sequenceid=8, filesize=63.8 K 2023-02-19 12:58:27,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc, entries=34, sequenceid=8, filesize=10.7 K 2023-02-19 12:58:27,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2, entries=32, sequenceid=8, filesize=10.4 K 2023-02-19 12:58:27,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 60f6a18c9ce621af4e18258a265d998f in 188ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:27,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for f3edc0852f9dc085eb7e7bda3e53af28 in 188ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:27,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 9073362bb9d4cd684678e7eb8e65b9f1 in 186ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:27,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676811504468' 2023-02-19 12:58:27,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676811504468' 2023-02-19 12:58:27,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676811504468' 2023-02-19 12:58:27,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:27,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:27,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:27,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:27,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:27,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:27,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:27,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:27,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:27,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:27,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:27,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:27,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:27,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:27,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:27,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:27,688 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:27,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e7b05a6d17f916f82251644d3eb30cbb 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-19 12:58:27,689 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811507688"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811507688"}]},"ts":"1676811507688"} 2023-02-19 12:58:27,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:27,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:27,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:27,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:27,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:27,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:27,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:27,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ac89ddd19c2efa260abf3eee99ed5c81 1/1 column families, dataSize=1.44 KB heapSize=3.34 KB 2023-02-19 12:58:27,691 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:27,691 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811507691"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811507691"}]},"ts":"1676811507691"} 2023-02-19 12:58:27,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:27,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:27,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:27,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:27,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:27,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:27,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:27,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 51f07ae8ab83176a1dfeaae89bd6da17 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-19 12:58:27,695 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:27,695 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811507695"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811507695"}]},"ts":"1676811507695"} 2023-02-19 12:58:27,704 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=53 2023-02-19 12:58:27,704 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=53, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769 in 358 msec 2023-02-19 12:58:27,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-02-19 12:58:27,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 376 msec 2023-02-19 12:58:27,710 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=50 2023-02-19 12:58:27,710 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 390 msec 2023-02-19 12:58:27,710 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=50, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 379 msec 2023-02-19 12:58:27,711 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 395 msec 2023-02-19 12:58:27,711 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 396 msec 2023-02-19 12:58:27,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:27,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:27,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:27,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/.tmp/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:27,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/.tmp/cf/7ed4e7cce591478da79ea8e1ed713797 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:27,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797, entries=36, sequenceid=8, filesize=11.0 K 2023-02-19 12:58:27,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for e7b05a6d17f916f82251644d3eb30cbb in 115ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:27,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:27,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:27,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:27,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:27,812 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:27,812 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811507812"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811507812"}]},"ts":"1676811507812"} 2023-02-19 12:58:27,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=52 2023-02-19 12:58:27,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=52, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 489 msec 2023-02-19 12:58:27,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 502 msec 2023-02-19 12:58:28,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-19 12:58:28,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/.tmp/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:28,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/.tmp/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:28,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/.tmp/cf/d1a1507f32ed4c32a17152c28671d734 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:28,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/.tmp/cf/a3f7fdeb36f24f4199df01d17ca229de as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:28,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734, entries=30, sequenceid=8, filesize=10.0 K 2023-02-19 12:58:28,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de, entries=22, sequenceid=8, filesize=8.6 K 2023-02-19 12:58:28,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 51f07ae8ab83176a1dfeaae89bd6da17 in 517ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:28,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.44 KB/1472, heapSize ~3.33 KB/3408, currentSize=0 B/0 for ac89ddd19c2efa260abf3eee99ed5c81 in 523ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:28,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:28,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:28,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:28,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:28,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:28,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:28,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:28,222 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:28,222 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811508222"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811508222"}]},"ts":"1676811508222"} 2023-02-19 12:58:28,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:28,224 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:28,224 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811508224"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811508224"}]},"ts":"1676811508224"} 2023-02-19 12:58:28,226 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=54 2023-02-19 12:58:28,226 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=54, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 898 msec 2023-02-19 12:58:28,228 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 912 msec 2023-02-19 12:58:28,229 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=51 2023-02-19 12:58:28,229 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=51, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737 in 904 msec 2023-02-19 12:58:28,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=49 2023-02-19 12:58:28,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 915 msec 2023-02-19 12:58:28,238 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811508237"}]},"ts":"1676811508237"} 2023-02-19 12:58:28,239 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:28,241 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:28,244 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 939 msec 2023-02-19 12:58:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-19 12:58:28,814 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 49 completed 2023-02-19 12:58:28,815 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:28,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811508815 (current time:1676811508815). 2023-02-19 12:58:28,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:28,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x6c3798cb to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:28,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d25cd94, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:28,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:28,823 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50390, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c3798cb to 127.0.0.1:60853 2023-02-19 12:58:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:28,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:28,829 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:28,830 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE; timeout=600000, timestamp=1676812108830 2023-02-19 12:58:28,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:28,830 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676811504468 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 2023-02-19 12:58:28,831 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:28,831 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:28,848 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,850 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:28,851 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:28,861 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de] hfiles 2023-02-19 12:58:28,861 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:28,862 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2] hfiles 2023-02-19 12:58:28,862 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc] hfiles 2023-02-19 12:58:28,862 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:28,862 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:28,866 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797] hfiles 2023-02-19 12:58:28,867 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:28,867 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734] hfiles 2023-02-19 12:58:28,867 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306] hfiles 2023-02-19 12:58:28,867 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:28,867 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a] hfiles 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:28,870 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:28,947 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-19 12:58:29,007 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-19 12:58:29,008 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-19 12:58:29,081 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:29,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:29,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' is still in progress! 2023-02-19 12:58:29,083 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-19 12:58:29,319 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:29,319 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676811504468 2023-02-19 12:58:29,320 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811504468 2023-02-19 12:58:29,355 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811504468 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/snaptb0-1676811504468 2023-02-19 12:58:29,383 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676811504468 of table testRestoreSnapshot-1676811504468 completed 2023-02-19 12:58:29,384 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811504468 2023-02-19 12:58:29,384 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811504468 2023-02-19 12:58:29,384 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676811504468 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 at 1676811508831Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } at 1676811508848 (+17 ms)Consolidate snapshot: snaptb0-1676811504468 at 1676811509319 (+471 ms)Loading Region manifests for snaptb0-1676811504468 at 1676811509319Writing data manifest for snaptb0-1676811504468 at 1676811509326 (+7 ms)Verifying snapshot: snaptb0-1676811504468 at 1676811509338 (+12 ms)Snapshot snaptb0-1676811504468 of table testRestoreSnapshot-1676811504468 completed at 1676811509383 (+45 ms) 2023-02-19 12:58:29,385 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:29,387 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE in 557 msec 2023-02-19 12:58:29,583 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:29,585 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:29,585 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676811504468 2023-02-19 12:58:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:29,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-19 12:58:29,590 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676811504468 2023-02-19 12:58:29,590 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811509590"}]},"ts":"1676811509590"} 2023-02-19 12:58:29,592 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:29,598 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:29,598 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:29,600 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:29,600 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:29,601 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:29,601 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:29,601 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:29,601 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:29,601 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:29,601 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:29,601 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:29,602 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:29,602 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:29,602 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:29,601 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:29,602 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:29,602 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:29,602 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:29,603 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:29,603 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:29,752 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:29,752 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:29,752 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:29,752 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811509752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811509752"}]},"ts":"1676811509752"} 2023-02-19 12:58:29,752 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:29,752 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811509752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811509752"}]},"ts":"1676811509752"} 2023-02-19 12:58:29,752 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:29,752 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811509752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811509752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811509752"}]},"ts":"1676811509752"} 2023-02-19 12:58:29,752 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:29,752 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811509752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811509752"}]},"ts":"1676811509752"} 2023-02-19 12:58:29,753 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811509752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811509752"}]},"ts":"1676811509752"} 2023-02-19 12:58:29,754 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=66, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:29,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=68, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:29,756 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=64, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:29,757 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=65, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:29,758 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=67, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:29,758 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:29,759 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811509758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811509758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811509758"}]},"ts":"1676811509758"} 2023-02-19 12:58:29,762 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=69, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:29,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-19 12:58:29,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:29,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:29,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:29,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:29,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:29,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:29,912 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,914 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:29,914 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:29,914 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:29,914 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,915 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,916 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:29,916 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:29,916 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:29,916 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:29,916 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:29,916 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:29,926 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:29,928 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:29,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,931 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:29,931 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:29,931 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:29,931 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:29,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:29,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68520017, jitterRate=0.0210278183221817}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:29,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:29,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:29,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=74, masterSystemTime=1676811509906 2023-02-19 12:58:29,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:29,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61351727, jitterRate=-0.08578802645206451}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:29,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:29,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66275076, jitterRate=-0.01242440938949585}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:29,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:29,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=72, masterSystemTime=1676811509908 2023-02-19 12:58:29,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=73, masterSystemTime=1676811509910 2023-02-19 12:58:29,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:29,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:29,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:29,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:29,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:29,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,943 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:29,943 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509943"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811509943"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811509943"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811509943"}]},"ts":"1676811509943"} 2023-02-19 12:58:29,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:29,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:29,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:29,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:29,944 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:29,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:29,944 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811509944"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811509944"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811509944"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811509944"}]},"ts":"1676811509944"} 2023-02-19 12:58:29,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:29,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:29,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:29,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:29,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:29,946 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,946 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:29,946 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509946"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811509946"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811509946"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811509946"}]},"ts":"1676811509946"} 2023-02-19 12:58:29,947 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:29,948 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:29,948 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,948 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:29,949 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,950 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:29,950 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:29,950 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=67 2023-02-19 12:58:29,951 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=67, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769 in 188 msec 2023-02-19 12:58:29,951 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:29,951 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:29,951 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:29,952 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:29,952 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=64 2023-02-19 12:58:29,953 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=64, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 192 msec 2023-02-19 12:58:29,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 352 msec 2023-02-19 12:58:29,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=65 2023-02-19 12:58:29,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=65, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737 in 192 msec 2023-02-19 12:58:29,955 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 355 msec 2023-02-19 12:58:29,956 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 356 msec 2023-02-19 12:58:29,958 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:29,958 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:29,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,961 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:29,961 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:29,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,962 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:29,962 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:29,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:29,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65115863, jitterRate=-0.029698029160499573}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:29,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:29,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:29,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=70, masterSystemTime=1676811509906 2023-02-19 12:58:29,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:29,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73852047, jitterRate=0.10048125684261322}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:29,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:29,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63810790, jitterRate=-0.049145132303237915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:29,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:29,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=71, masterSystemTime=1676811509908 2023-02-19 12:58:29,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:29,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:29,972 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:29,972 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509972"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811509972"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811509972"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811509972"}]},"ts":"1676811509972"} 2023-02-19 12:58:29,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=75, masterSystemTime=1676811509910 2023-02-19 12:58:29,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:29,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:29,974 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:29,974 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811509974"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811509974"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811509974"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811509974"}]},"ts":"1676811509974"} 2023-02-19 12:58:29,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:29,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:29,976 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:29,976 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811509975"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811509975"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811509975"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811509975"}]},"ts":"1676811509975"} 2023-02-19 12:58:29,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=66 2023-02-19 12:58:29,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=66, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 220 msec 2023-02-19 12:58:29,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=68 2023-02-19 12:58:29,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=68, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 221 msec 2023-02-19 12:58:29,980 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 380 msec 2023-02-19 12:58:29,981 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=69 2023-02-19 12:58:29,981 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=69, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 218 msec 2023-02-19 12:58:29,981 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 382 msec 2023-02-19 12:58:29,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=69, resume processing ppid=63 2023-02-19 12:58:29,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 383 msec 2023-02-19 12:58:29,984 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811509984"}]},"ts":"1676811509984"} 2023-02-19 12:58:29,985 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:29,987 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676811504468' was successfully enabled. 2023-02-19 12:58:29,990 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676811504468 in 403 msec 2023-02-19 12:58:30,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-19 12:58:30,342 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 63 completed 2023-02-19 12:58:30,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:30,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:30,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:30,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:30,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:30,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:30,370 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676811504468 2023-02-19 12:58:30,370 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:30,371 INFO [Listener at localhost/43399] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-19 12:58:30,634 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:30,697 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-19 12:58:30,698 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-19 12:58:30,743 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:30,917 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:31,060 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:31,218 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:32,613 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(139): Connect 0x290fa1b6 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:32,619 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b77b3f7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:32,621 DEBUG [hconnection-0x20ceba7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:32,625 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50394, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:32,630 DEBUG [hconnection-0x20ceba7a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:32,633 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:32,765 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:32,769 DEBUG [hconnection-0x20ceba7a-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:32,774 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:32,856 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:32,977 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:33,094 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:33,202 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:34,375 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:34,375 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:34,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:34,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-19 12:58:34,381 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811514380"}]},"ts":"1676811514380"} 2023-02-19 12:58:34,382 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:34,384 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:34,385 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:34,387 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:34,387 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:34,388 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:34,388 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:34,388 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:34,389 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:34,389 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:34,389 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811514388"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811514388"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811514388"}]},"ts":"1676811514388"} 2023-02-19 12:58:34,389 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:34,389 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811514389"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811514389"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811514389"}]},"ts":"1676811514389"} 2023-02-19 12:58:34,389 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:34,389 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811514389"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811514389"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811514389"}]},"ts":"1676811514389"} 2023-02-19 12:58:34,389 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:34,389 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811514389"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811514389"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811514389"}]},"ts":"1676811514389"} 2023-02-19 12:58:34,390 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811514389"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811514389"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811514389"}]},"ts":"1676811514389"} 2023-02-19 12:58:34,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=81, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:34,392 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=79, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:34,393 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=78, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:34,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=82, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:34,395 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=80, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:34,396 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:34,398 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:34,398 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811514398"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811514398"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811514398"}]},"ts":"1676811514398"} 2023-02-19 12:58:34,406 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=77, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:34,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:34,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:34,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:34,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:34,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:34,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:34,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 51f07ae8ab83176a1dfeaae89bd6da17 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-19 12:58:34,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:34,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:34,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:34,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:34,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:34,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:34,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9073362bb9d4cd684678e7eb8e65b9f1 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-19 12:58:34,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:34,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:34,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:34,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:34,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:34,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:34,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ac89ddd19c2efa260abf3eee99ed5c81 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-19 12:58:34,574 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 2023-02-19 12:58:34,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 2023-02-19 12:58:34,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 2023-02-19 12:58:34,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/.tmp/cf/3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:34,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/.tmp/cf/8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:34,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/.tmp/cf/3809b72d50754a1f92510fd72cc59c86 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:34,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/.tmp/cf/8bdcfc8bf7444dc9ae76249f5c286818 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:34,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86, entries=25, sequenceid=15, filesize=9.1 K 2023-02-19 12:58:34,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 51f07ae8ab83176a1dfeaae89bd6da17 in 62ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:34,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818, entries=23, sequenceid=15, filesize=8.8 K 2023-02-19 12:58:34,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for ac89ddd19c2efa260abf3eee99ed5c81 in 64ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:34,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:34,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:34,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:34,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:34,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:34,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:34,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:34,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:34,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:34,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:34,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:34,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:34,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:34,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 60f6a18c9ce621af4e18258a265d998f 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-19 12:58:34,620 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:34,620 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811514620"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811514620"}]},"ts":"1676811514620"} 2023-02-19 12:58:34,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:34,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:34,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:34,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:34,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:34,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:34,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:34,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f3edc0852f9dc085eb7e7bda3e53af28 1/1 column families, dataSize=23.29 KB heapSize=50.31 KB 2023-02-19 12:58:34,625 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:34,625 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811514625"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811514625"}]},"ts":"1676811514625"} 2023-02-19 12:58:34,629 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=81 2023-02-19 12:58:34,629 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=81, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 233 msec 2023-02-19 12:58:34,631 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 244 msec 2023-02-19 12:58:34,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=78 2023-02-19 12:58:34,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-19 12:58:34,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=78, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,39199,1676811501737 in 236 msec 2023-02-19 12:58:34,633 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 246 msec 2023-02-19 12:58:34,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 2023-02-19 12:58:34,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 2023-02-19 12:58:34,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/.tmp/cf/f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:34,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/.tmp/cf/f9a520182acc4e20a59ab4fedc12f191 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:34,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=23.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/.tmp/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:34,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191, entries=29, sequenceid=15, filesize=9.8 K 2023-02-19 12:58:34,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 60f6a18c9ce621af4e18258a265d998f in 41ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:34,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/.tmp/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:34,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0, entries=356, sequenceid=15, filesize=65.5 K 2023-02-19 12:58:34,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:34,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:34,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:34,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.29 KB/23848, heapSize ~50.30 KB/51504, currentSize=0 B/0 for f3edc0852f9dc085eb7e7bda3e53af28 in 46ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:34,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:34,673 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:34,674 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811514673"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811514673"}]},"ts":"1676811514673"} 2023-02-19 12:58:34,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=77 2023-02-19 12:58:34,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=77, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 269 msec 2023-02-19 12:58:34,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:34,679 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 292 msec 2023-02-19 12:58:34,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:34,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:34,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:34,681 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:34,681 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811514681"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811514681"}]},"ts":"1676811514681"} 2023-02-19 12:58:34,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=82 2023-02-19 12:58:34,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=82, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 288 msec 2023-02-19 12:58:34,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 299 msec 2023-02-19 12:58:35,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/.tmp/cf/d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:35,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/.tmp/cf/d7a343597ad64954a497481cd3e01d4d as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:35,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d, entries=32, sequenceid=15, filesize=10.4 K 2023-02-19 12:58:35,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 9073362bb9d4cd684678e7eb8e65b9f1 in 464ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:35,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:35,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:35,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:35,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:35,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:35,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:35,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:35,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:35,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:35,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:35,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e7b05a6d17f916f82251644d3eb30cbb 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-19 12:58:35,020 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:35,020 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811515020"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515020"}]},"ts":"1676811515020"} 2023-02-19 12:58:35,026 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=80 2023-02-19 12:58:35,026 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=80, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,45077,1676811501769 in 627 msec 2023-02-19 12:58:35,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 641 msec 2023-02-19 12:58:35,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 2023-02-19 12:58:35,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/.tmp/cf/3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:35,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/.tmp/cf/3120ab30e9cd410aa1c81515aec2e8ba as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:35,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba, entries=35, sequenceid=15, filesize=10.9 K 2023-02-19 12:58:35,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for e7b05a6d17f916f82251644d3eb30cbb in 47ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:35,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:35,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:35,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:35,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:35,078 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:35,078 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811515078"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515078"}]},"ts":"1676811515078"} 2023-02-19 12:58:35,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=79 2023-02-19 12:58:35,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=79, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 688 msec 2023-02-19 12:58:35,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=79, resume processing ppid=76 2023-02-19 12:58:35,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 696 msec 2023-02-19 12:58:35,084 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811515084"}]},"ts":"1676811515084"} 2023-02-19 12:58:35,085 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:35,088 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:35,090 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 714 msec 2023-02-19 12:58:35,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-19 12:58:35,133 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 76 completed 2023-02-19 12:58:35,134 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811515134 (current time:1676811515134). 2023-02-19 12:58:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x41d1ee39 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:35,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@11f0d708, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:35,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:35,142 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:35,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41d1ee39 to 127.0.0.1:60853 2023-02-19 12:58:35,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:35,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:35,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:35,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:35,146 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:35,147 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE; timeout=600000, timestamp=1676812115147 2023-02-19 12:58:35,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:35,147 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676811504468 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,147 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:35,148 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:35,161 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:35,162 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,162 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,162 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,162 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,162 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,163 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,164 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,164 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191] hfiles 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734] hfiles 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0] hfiles 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d] hfiles 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:35,167 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:35,168 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:35,169 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de] hfiles 2023-02-19 12:58:35,169 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:35,169 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:35,169 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797] hfiles 2023-02-19 12:58:35,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:35,170 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:35,184 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a] hfiles 2023-02-19 12:58:35,184 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 2023-02-19 12:58:35,184 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:35,184 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:35,184 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 2023-02-19 12:58:35,185 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:35,204 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:35,204 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676811504468 2023-02-19 12:58:35,205 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb1-1676811504468 2023-02-19 12:58:35,234 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb1-1676811504468 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/snaptb1-1676811504468 2023-02-19 12:58:35,267 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676811504468 of table testRestoreSnapshot-1676811504468 completed 2023-02-19 12:58:35,267 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb1-1676811504468 2023-02-19 12:58:35,268 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb1-1676811504468 2023-02-19 12:58:35,268 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676811504468 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 at 1676811515147Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } at 1676811515161 (+14 ms)Consolidate snapshot: snaptb1-1676811504468 at 1676811515204 (+43 ms)Loading Region manifests for snaptb1-1676811504468 at 1676811515204Writing data manifest for snaptb1-1676811504468 at 1676811515208 (+4 ms)Verifying snapshot: snaptb1-1676811504468 at 1676811515217 (+9 ms)Snapshot snaptb1-1676811504468 of table testRestoreSnapshot-1676811504468 completed at 1676811515267 (+50 ms) 2023-02-19 12:58:35,269 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:35,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE in 124 msec 2023-02-19 12:58:35,398 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:35,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:35,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:35,414 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-19 12:58:35,416 INFO [Listener at localhost/43399] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,416 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:35,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811515416 (current time:1676811515416). 2023-02-19 12:58:35,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:35,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x10406316 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:35,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31aa0449, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:35,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:35,431 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52274, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:35,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10406316 to 127.0.0.1:60853 2023-02-19 12:58:35,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:35,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:35,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:35,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:35,435 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:35,436 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE; timeout=600000, timestamp=1676812115436 2023-02-19 12:58:35,436 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676811504468-1676811515415 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:35,436 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:35,437 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:35,453 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:35,459 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,459 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,459 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,459 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,459 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,460 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,460 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,460 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,461 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,461 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,461 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,461 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,461 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:35,461 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:35,466 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0] hfiles 2023-02-19 12:58:35,466 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:35,466 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:35,472 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de] hfiles 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797] hfiles 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d] hfiles 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:35,501 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:35,502 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734] hfiles 2023-02-19 12:58:35,502 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:35,503 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:35,503 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x290fa1b6 to 127.0.0.1:60853 2023-02-19 12:58:35,503 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:35,506 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191] hfiles 2023-02-19 12:58:35,506 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:35,506 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a] hfiles 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:35,516 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 2023-02-19 12:58:35,517 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 2023-02-19 12:58:35,517 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:35,517 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 2023-02-19 12:58:35,517 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:35,549 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:35,549 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,550 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,579 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676811504468-1676811515415 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,598 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676811504468-1676811515415 of table testRestoreSnapshot-1676811504468 completed 2023-02-19 12:58:35,599 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,599 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,599 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676811504468-1676811515415 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 at 1676811515436Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } at 1676811515453 (+17 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676811504468-1676811515415 at 1676811515549 (+96 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676811504468-1676811515415 at 1676811515549Writing data manifest for hbase-failsafe-snaptb0-1676811504468-1676811515415 at 1676811515554 (+5 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676811504468-1676811515415 at 1676811515566 (+12 ms)Snapshot hbase-failsafe-snaptb0-1676811504468-1676811515415 of table testRestoreSnapshot-1676811504468 completed at 1676811515598 (+32 ms) 2023-02-19 12:58:35,601 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:35,602 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE in 167 msec 2023-02-19 12:58:35,687 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676811504468-1676811515415 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:35,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "snaptb0-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811508815 type: DISABLED version: 2 ttl: 0 ) 2023-02-19 12:58:35,712 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676811504468 as table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-19 12:58:35,727 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000002 2023-02-19 12:58:35,728 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:35,728 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000002 2023-02-19 12:58:35,729 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:35,733 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811508815 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:35,734 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,742 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,749 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 2023-02-19 12:58:35,751 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 2023-02-19 12:58:35,753 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 2023-02-19 12:58:35,754 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 2023-02-19 12:58:35,756 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 2023-02-19 12:58:35,757 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 2023-02-19 12:58:35,757 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:35,758 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:35,758 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:35,758 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:35,758 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:35,758 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:35,763 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:35,763 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:35,763 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:35,763 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:35,763 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:35,763 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:35,764 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811508815 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:35,765 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811515765"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:35,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811515765"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:35,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811515765"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:35,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811515765"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:35,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811515765"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:35,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811515765"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:35,769 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811515766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515766"}]},"ts":"1676811515766"} 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811515766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515766"}]},"ts":"1676811515766"} 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811515766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515766"}]},"ts":"1676811515766"} 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811515766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515766"}]},"ts":"1676811515766"} 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811515766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515766"}]},"ts":"1676811515766"} 2023-02-19 12:58:35,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811515766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811515766"}]},"ts":"1676811515766"} 2023-02-19 12:58:35,772 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-19 12:58:35,772 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-19 12:58:35,772 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:35,772 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676811504468 completed! 2023-02-19 12:58:35,774 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "snaptb0-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811508815 type: DISABLED version: 2 ttl: 0 ) in 69 msec 2023-02-19 12:58:35,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-19 12:58:35,968 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676811504468, procId: 91 completed 2023-02-19 12:58:35,968 INFO [Listener at localhost/43399] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,972 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676811504468-1676811515415" 2023-02-19 12:58:35,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676811504468-1676811515415 2023-02-19 12:58:35,976 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,977 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-19 12:58:35,981 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676811504468 2023-02-19 12:58:35,981 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811515981"}]},"ts":"1676811515981"} 2023-02-19 12:58:35,982 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:35,984 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:35,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:35,990 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:35,990 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:35,990 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:35,990 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:35,990 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:35,991 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:35,991 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:35,991 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:35,991 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:35,991 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:35,992 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:35,992 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:36,141 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:36,141 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:36,141 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:36,141 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:36,141 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:36,144 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:36,144 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:36,145 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811516144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811516144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811516144"}]},"ts":"1676811516144"} 2023-02-19 12:58:36,144 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:36,145 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811516144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811516144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811516144"}]},"ts":"1676811516144"} 2023-02-19 12:58:36,144 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:36,144 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:36,145 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811516144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811516144"}]},"ts":"1676811516144"} 2023-02-19 12:58:36,145 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811516144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811516144"}]},"ts":"1676811516144"} 2023-02-19 12:58:36,145 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516144"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811516144"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811516144"}]},"ts":"1676811516144"} 2023-02-19 12:58:36,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=93, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:36,147 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=98, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:36,148 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=97, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:36,149 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=94, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:36,150 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=96, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:36,151 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:36,151 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516150"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811516150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811516150"}]},"ts":"1676811516150"} 2023-02-19 12:58:36,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:36,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-19 12:58:36,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:36,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:36,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:36,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:36,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,303 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,303 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,303 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,304 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:36,304 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:36,304 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:36,304 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:36,304 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:36,304 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:36,305 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:36,305 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:36,305 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:36,310 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:36,311 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:36,311 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:36,311 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:36,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,313 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:36,313 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:36,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:36,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:36,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70962038, jitterRate=0.05741676688194275}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:36,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:36,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68490995, jitterRate=0.020595356822013855}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:36,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:36,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=104, masterSystemTime=1676811516299 2023-02-19 12:58:36,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=99, masterSystemTime=1676811516299 2023-02-19 12:58:36,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:36,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:36,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:36,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:36,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:36,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74109663, jitterRate=0.10432003438472748}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:36,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:36,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:36,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=102, masterSystemTime=1676811516300 2023-02-19 12:58:36,321 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:36,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:36,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:36,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:36,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:36,321 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516321"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811516321"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811516321"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811516321"}]},"ts":"1676811516321"} 2023-02-19 12:58:36,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:36,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,322 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:36,322 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811516322"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811516322"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811516322"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811516322"}]},"ts":"1676811516322"} 2023-02-19 12:58:36,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:36,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:36,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:36,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:36,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:36,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,323 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,323 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:36,324 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516323"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811516323"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811516323"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811516323"}]},"ts":"1676811516323"} 2023-02-19 12:58:36,325 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:36,325 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:36,325 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:36,326 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-19 12:58:36,326 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737 in 170 msec 2023-02-19 12:58:36,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=93 2023-02-19 12:58:36,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=93, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 178 msec 2023-02-19 12:58:36,328 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 338 msec 2023-02-19 12:58:36,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=94 2023-02-19 12:58:36,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=94, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769 in 177 msec 2023-02-19 12:58:36,328 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,328 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,328 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 342 msec 2023-02-19 12:58:36,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 343 msec 2023-02-19 12:58:36,331 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:36,331 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:36,331 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:36,331 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:36,332 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:36,332 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:36,334 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:36,334 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:36,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,336 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:36,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,336 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:36,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,338 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:36,338 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:36,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:36,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59209731, jitterRate=-0.11770625412464142}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:36,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:36,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:36,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=103, masterSystemTime=1676811516299 2023-02-19 12:58:36,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61277716, jitterRate=-0.08689087629318237}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:36,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:36,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:36,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=100, masterSystemTime=1676811516299 2023-02-19 12:58:36,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:36,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:36,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61578176, jitterRate=-0.0824136734008789}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:36,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:36,344 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:36,344 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516344"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811516344"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811516344"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811516344"}]},"ts":"1676811516344"} 2023-02-19 12:58:36,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=101, masterSystemTime=1676811516300 2023-02-19 12:58:36,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:36,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:36,345 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:36,345 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811516345"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811516345"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811516345"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811516345"}]},"ts":"1676811516345"} 2023-02-19 12:58:36,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:36,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:36,347 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:36,347 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811516347"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811516347"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811516347"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811516347"}]},"ts":"1676811516347"} 2023-02-19 12:58:36,349 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=96 2023-02-19 12:58:36,349 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=96, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696 in 196 msec 2023-02-19 12:58:36,350 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=98 2023-02-19 12:58:36,350 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=98, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 201 msec 2023-02-19 12:58:36,351 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 361 msec 2023-02-19 12:58:36,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=97 2023-02-19 12:58:36,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=97, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769 in 201 msec 2023-02-19 12:58:36,351 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 362 msec 2023-02-19 12:58:36,357 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=92 2023-02-19 12:58:36,357 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 363 msec 2023-02-19 12:58:36,357 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811516357"}]},"ts":"1676811516357"} 2023-02-19 12:58:36,358 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:36,360 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676811504468' was successfully enabled. 2023-02-19 12:58:36,362 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676811504468 in 384 msec 2023-02-19 12:58:36,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-19 12:58:36,732 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 92 completed 2023-02-19 12:58:36,733 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(139): Connect 0x7c1707f4 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:36,739 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29e7c4cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:36,740 DEBUG [hconnection-0x3e907963-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:36,744 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52282, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:36,748 DEBUG [hconnection-0x3e907963-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:36,751 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41270, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:36,857 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:36,922 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:36,925 DEBUG [hconnection-0x3e907963-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:36,928 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39842, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:37,030 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:37,122 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:37,204 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:38,103 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,103 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-19 12:58:38,107 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811518107"}]},"ts":"1676811518107"} 2023-02-19 12:58:38,108 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:38,110 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:38,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:38,112 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:38,112 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:38,113 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:38,113 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:38,113 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:38,113 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:38,114 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:38,114 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811518113"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811518113"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811518113"}]},"ts":"1676811518113"} 2023-02-19 12:58:38,114 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:38,114 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:38,114 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:38,114 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518114"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811518114"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811518114"}]},"ts":"1676811518114"} 2023-02-19 12:58:38,114 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811518113"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811518113"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811518113"}]},"ts":"1676811518113"} 2023-02-19 12:58:38,114 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518114"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811518114"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811518114"}]},"ts":"1676811518114"} 2023-02-19 12:58:38,114 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518113"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811518113"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811518113"}]},"ts":"1676811518113"} 2023-02-19 12:58:38,115 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=111, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:38,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=106, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:38,119 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=110, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:38,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:38,121 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=107, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:38,121 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:38,122 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:38,122 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518122"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811518122"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811518122"}]},"ts":"1676811518122"} 2023-02-19 12:58:38,124 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=108, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:38,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:38,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:38,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:38,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:38,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:38,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:38,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:38,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:38,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:38,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:38,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:38,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:38,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:38,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:38,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:38,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:38,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:38,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:38,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-19 12:58:38,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-19 12:58:38,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-19 12:58:38,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:38,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:38,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:38,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:38,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:38,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:38,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:38,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:38,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:38,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:38,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:38,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:38,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:38,286 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:38,286 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518286"}]},"ts":"1676811518286"} 2023-02-19 12:58:38,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:38,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:38,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:38,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:38,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:38,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:38,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:38,287 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:38,287 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518287"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518287"}]},"ts":"1676811518287"} 2023-02-19 12:58:38,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:38,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:38,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:38,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:38,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:38,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:38,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:38,289 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:38,289 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811518289"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518289"}]},"ts":"1676811518289"} 2023-02-19 12:58:38,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-19 12:58:38,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-19 12:58:38,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=108 2023-02-19 12:58:38,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=108, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737 in 165 msec 2023-02-19 12:58:38,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:38,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:38,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:38,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:38,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-19 12:58:38,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=110 2023-02-19 12:58:38,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=110, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769 in 171 msec 2023-02-19 12:58:38,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:38,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:38,293 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=106 2023-02-19 12:58:38,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 181 msec 2023-02-19 12:58:38,293 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=106, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 175 msec 2023-02-19 12:58:38,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:38,294 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 181 msec 2023-02-19 12:58:38,294 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:38,294 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518294"}]},"ts":"1676811518294"} 2023-02-19 12:58:38,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:38,294 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 182 msec 2023-02-19 12:58:38,294 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:38,294 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811518294"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518294"}]},"ts":"1676811518294"} 2023-02-19 12:58:38,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:38,295 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:38,295 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518295"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518295"}]},"ts":"1676811518295"} 2023-02-19 12:58:38,298 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-19 12:58:38,298 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696 in 175 msec 2023-02-19 12:58:38,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=111 2023-02-19 12:58:38,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=111, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 181 msec 2023-02-19 12:58:38,300 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=107 2023-02-19 12:58:38,300 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=107, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769 in 176 msec 2023-02-19 12:58:38,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 187 msec 2023-02-19 12:58:38,300 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 187 msec 2023-02-19 12:58:38,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=105 2023-02-19 12:58:38,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 189 msec 2023-02-19 12:58:38,302 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811518302"}]},"ts":"1676811518302"} 2023-02-19 12:58:38,303 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:38,305 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:38,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 202 msec 2023-02-19 12:58:38,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-19 12:58:38,358 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 105 completed 2023-02-19 12:58:38,366 INFO [Listener at localhost/43399] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,367 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:38,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811518367 (current time:1676811518367). 2023-02-19 12:58:38,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:38,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x0d0f408b to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:38,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f78ed17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:38,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:38,375 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:38,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d0f408b to 127.0.0.1:60853 2023-02-19 12:58:38,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:38,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:38,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:38,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:38,379 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:38,380 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE; timeout=600000, timestamp=1676812118380 2023-02-19 12:58:38,380 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676811504468-1676811518366 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:38,380 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:38,380 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:38,390 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:38,393 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,393 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,393 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,393 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,396 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,397 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc] hfiles 2023-02-19 12:58:38,397 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:38,397 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306] hfiles 2023-02-19 12:58:38,397 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:38,399 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:38,399 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:38,400 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de] hfiles 2023-02-19 12:58:38,400 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:38,401 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797] hfiles 2023-02-19 12:58:38,401 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:38,403 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2] hfiles 2023-02-19 12:58:38,403 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:38,404 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734] hfiles 2023-02-19 12:58:38,404 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a] hfiles 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:38,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:38,428 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:38,428 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,429 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,460 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676811504468-1676811518366 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,477 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676811504468-1676811518366 of table testRestoreSnapshot-1676811504468 completed 2023-02-19 12:58:38,477 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,477 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,477 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676811504468-1676811518366 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 at 1676811518380Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } at 1676811518390 (+10 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676811504468-1676811518366 at 1676811518428 (+38 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676811504468-1676811518366 at 1676811518429 (+1 ms)Writing data manifest for hbase-failsafe-emptySnaptb-1676811504468-1676811518366 at 1676811518436 (+7 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676811504468-1676811518366 at 1676811518447 (+11 ms)Snapshot hbase-failsafe-emptySnaptb-1676811504468-1676811518366 of table testRestoreSnapshot-1676811504468 completed at 1676811518477 (+30 ms) 2023-02-19 12:58:38,479 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:38,481 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE in 101 msec 2023-02-19 12:58:38,630 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:38,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:38,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676811504468-1676811518366 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:38,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "emptySnaptb-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811505633 type: DISABLED version: 2 ttl: 0 ) 2023-02-19 12:58:38,637 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676811504468 as table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-19 12:58:38,650 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000003 2023-02-19 12:58:38,651 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000002 2023-02-19 12:58:38,651 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000003 2023-02-19 12:58:38,652 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:38,654 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811505633 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:38,654 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,661 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,664 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:38,665 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:38,666 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:38,668 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:38,669 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:38,670 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:38,670 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:38,670 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:38,670 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:38,670 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:38,670 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:38,670 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:38,675 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:38,675 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:38,676 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:38,676 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:38,676 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:38,676 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:38,677 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811505633 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:38,678 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811518678"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:38,678 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811518678"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:38,678 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811518678"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:38,678 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811518678"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:38,679 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811518678"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:38,679 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811518678"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:38,681 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-19 12:58:38,681 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:38,681 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518679"}]},"ts":"1676811518679"} 2023-02-19 12:58:38,681 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811518679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518679"}]},"ts":"1676811518679"} 2023-02-19 12:58:38,682 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518679"}]},"ts":"1676811518679"} 2023-02-19 12:58:38,682 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518679"}]},"ts":"1676811518679"} 2023-02-19 12:58:38,682 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811518679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518679"}]},"ts":"1676811518679"} 2023-02-19 12:58:38,682 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811518679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811518679"}]},"ts":"1676811518679"} 2023-02-19 12:58:38,684 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-19 12:58:38,684 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-19 12:58:38,684 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:38,684 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676811504468 completed! 2023-02-19 12:58:38,686 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "emptySnaptb-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811505633 type: DISABLED version: 2 ttl: 0 ) in 49 msec 2023-02-19 12:58:38,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-19 12:58:38,890 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676811504468, procId: 119 completed 2023-02-19 12:58:38,890 INFO [Listener at localhost/43399] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,891 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676811504468-1676811518366" 2023-02-19 12:58:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676811504468-1676811518366 2023-02-19 12:58:38,894 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,894 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-19 12:58:38,898 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676811504468 2023-02-19 12:58:38,898 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811518898"}]},"ts":"1676811518898"} 2023-02-19 12:58:38,899 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:38,901 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:38,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:38,904 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:38,904 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:38,904 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:38,904 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:38,904 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:38,905 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:38,905 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:38,905 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:38,905 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:38,905 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:38,906 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:38,906 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:39,055 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:39,055 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:39,055 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:39,055 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:39,055 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:39,060 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:39,060 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:39,060 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:39,060 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519060"}]},"ts":"1676811519060"} 2023-02-19 12:58:39,060 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519060"}]},"ts":"1676811519060"} 2023-02-19 12:58:39,060 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:39,060 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:39,060 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519060"}]},"ts":"1676811519060"} 2023-02-19 12:58:39,060 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519060"}]},"ts":"1676811519060"} 2023-02-19 12:58:39,060 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519060"}]},"ts":"1676811519060"} 2023-02-19 12:58:39,062 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=122, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:39,063 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=123, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:39,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=124, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:39,068 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=126, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:39,069 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:39,069 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519069"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519069"}]},"ts":"1676811519069"} 2023-02-19 12:58:39,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=125, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:39,071 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=121, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:39,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-19 12:58:39,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:39,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:39,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,221 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,222 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:39,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,223 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:39,223 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,223 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:39,223 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:39,223 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:39,223 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:39,223 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:39,224 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:39,224 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:39,225 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:39,225 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:39,225 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:39,225 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:39,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74119173, jitterRate=0.10446174442768097}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:39,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62239995, jitterRate=-0.07255180180072784}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:39,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:39,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:39,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71074417, jitterRate=0.059091344475746155}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:39,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:39,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=127, masterSystemTime=1676811519217 2023-02-19 12:58:39,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=129, masterSystemTime=1676811519219 2023-02-19 12:58:39,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=130, masterSystemTime=1676811519218 2023-02-19 12:58:39,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:39,232 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:39,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,232 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519232"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811519232"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811519232"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811519232"}]},"ts":"1676811519232"} 2023-02-19 12:58:39,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:39,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:39,233 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,233 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519233"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811519233"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811519233"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811519233"}]},"ts":"1676811519233"} 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,234 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,234 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,234 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519234"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811519234"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811519234"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811519234"}]},"ts":"1676811519234"} 2023-02-19 12:58:39,235 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,236 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,236 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:39,236 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:39,236 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:39,237 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:39,237 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:39,237 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:39,237 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:39,238 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:39,238 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:39,238 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:39,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,239 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:39,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=122 2023-02-19 12:58:39,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=122, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,41307,1676811501696 in 172 msec 2023-02-19 12:58:39,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,240 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:39,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59932333, jitterRate=-0.10693864524364471}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:39,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:39,244 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=124 2023-02-19 12:58:39,244 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=124, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 169 msec 2023-02-19 12:58:39,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=126 2023-02-19 12:58:39,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 338 msec 2023-02-19 12:58:39,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=126, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,45077,1676811501769 in 168 msec 2023-02-19 12:58:39,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=132, masterSystemTime=1676811519219 2023-02-19 12:58:39,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,245 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 343 msec 2023-02-19 12:58:39,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 343 msec 2023-02-19 12:58:39,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70020272, jitterRate=0.04338335990905762}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:39,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59907728, jitterRate=-0.10730528831481934}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:39,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:39,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:39,246 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:39,246 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519246"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811519246"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811519246"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811519246"}]},"ts":"1676811519246"} 2023-02-19 12:58:39,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=128, masterSystemTime=1676811519218 2023-02-19 12:58:39,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=131, masterSystemTime=1676811519217 2023-02-19 12:58:39,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,250 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:39,250 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519250"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811519250"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811519250"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811519250"}]},"ts":"1676811519250"} 2023-02-19 12:58:39,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,250 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=121 2023-02-19 12:58:39,251 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=121, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,39199,1676811501737 in 176 msec 2023-02-19 12:58:39,251 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:39,251 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519250"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811519250"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811519250"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811519250"}]},"ts":"1676811519250"} 2023-02-19 12:58:39,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 350 msec 2023-02-19 12:58:39,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=123 2023-02-19 12:58:39,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=123, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 189 msec 2023-02-19 12:58:39,254 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=125 2023-02-19 12:58:39,254 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=125, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 183 msec 2023-02-19 12:58:39,254 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 352 msec 2023-02-19 12:58:39,255 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=125, resume processing ppid=120 2023-02-19 12:58:39,255 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 353 msec 2023-02-19 12:58:39,256 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811519255"}]},"ts":"1676811519255"} 2023-02-19 12:58:39,257 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:39,258 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676811504468' was successfully enabled. 2023-02-19 12:58:39,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676811504468 in 365 msec 2023-02-19 12:58:39,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-19 12:58:39,650 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 120 completed 2023-02-19 12:58:39,650 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(139): Connect 0x2606f9a8 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:39,655 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58227bee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:39,657 DEBUG [hconnection-0x7bfe9d88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:39,659 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52296, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:39,662 DEBUG [hconnection-0x7bfe9d88-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:39,663 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39846, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:39,664 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:39,666 DEBUG [hconnection-0x7bfe9d88-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:39,669 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41274, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:39,670 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:39,672 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:39,675 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:39,678 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:39,685 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:39,685 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:39,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:39,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-19 12:58:39,689 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811519689"}]},"ts":"1676811519689"} 2023-02-19 12:58:39,690 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:39,693 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:39,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:39,695 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:39,695 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:39,695 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:39,696 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:39,696 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:39,696 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:39,696 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:39,696 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519696"}]},"ts":"1676811519696"} 2023-02-19 12:58:39,696 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519696"}]},"ts":"1676811519696"} 2023-02-19 12:58:39,697 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:39,697 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:39,697 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519697"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519697"}]},"ts":"1676811519697"} 2023-02-19 12:58:39,697 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519697"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519697"}]},"ts":"1676811519697"} 2023-02-19 12:58:39,697 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:39,698 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519697"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519697"}]},"ts":"1676811519697"} 2023-02-19 12:58:39,698 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=135, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:39,699 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=134, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:39,700 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=139, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:39,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=138, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:39,701 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:39,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=137, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:39,702 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:39,702 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519702"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811519702"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811519702"}]},"ts":"1676811519702"} 2023-02-19 12:58:39,704 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=136, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:39,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:39,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:39,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:39,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:39,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-19 12:58:39,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-19 12:58:39,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-19 12:58:39,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:39,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:39,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:39,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:39,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:39,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:39,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:39,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:39,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:39,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,860 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:39,860 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811519860"}]},"ts":"1676811519860"} 2023-02-19 12:58:39,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:39,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:39,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:39,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,862 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:39,862 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519862"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811519862"}]},"ts":"1676811519862"} 2023-02-19 12:58:39,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:39,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:39,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:39,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,864 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:39,864 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811519864"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811519864"}]},"ts":"1676811519864"} 2023-02-19 12:58:39,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-19 12:58:39,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:39,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:39,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-19 12:58:39,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:39,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:39,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-19 12:58:39,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:39,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:39,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:39,869 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=138 2023-02-19 12:58:39,869 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=138, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 162 msec 2023-02-19 12:58:39,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:39,869 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:39,869 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519869"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811519869"}]},"ts":"1676811519869"} 2023-02-19 12:58:39,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=134 2023-02-19 12:58:39,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=134, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,39199,1676811501737 in 166 msec 2023-02-19 12:58:39,870 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:39,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=139 2023-02-19 12:58:39,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:39,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=139, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,45077,1676811501769 in 166 msec 2023-02-19 12:58:39,870 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519870"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811519870"}]},"ts":"1676811519870"} 2023-02-19 12:58:39,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 175 msec 2023-02-19 12:58:39,873 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:39,873 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811519873"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811519873"}]},"ts":"1676811519873"} 2023-02-19 12:58:39,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 176 msec 2023-02-19 12:58:39,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 176 msec 2023-02-19 12:58:39,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=137 2023-02-19 12:58:39,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=136 2023-02-19 12:58:39,875 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=137, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 169 msec 2023-02-19 12:58:39,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=136, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,45077,1676811501769 in 168 msec 2023-02-19 12:58:39,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 181 msec 2023-02-19 12:58:39,875 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 181 msec 2023-02-19 12:58:39,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=135 2023-02-19 12:58:39,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=135, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,41307,1676811501696 in 176 msec 2023-02-19 12:58:39,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=135, resume processing ppid=133 2023-02-19 12:58:39,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 182 msec 2023-02-19 12:58:39,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811519878"}]},"ts":"1676811519878"} 2023-02-19 12:58:39,879 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:39,880 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:39,882 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 195 msec 2023-02-19 12:58:39,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-19 12:58:39,940 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 133 completed 2023-02-19 12:58:39,949 INFO [Listener at localhost/43399] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:39,950 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:39,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811519950 (current time:1676811519950). 2023-02-19 12:58:39,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:39,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x64924100 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:39,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38871769, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:39,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:39,958 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52310, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:39,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64924100 to 127.0.0.1:60853 2023-02-19 12:58:39,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:39,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:39,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:39,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:39,962 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:39,963 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE; timeout=600000, timestamp=1676812119963 2023-02-19 12:58:39,963 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676811504468-1676811519949 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 2023-02-19 12:58:39,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } 2023-02-19 12:58:39,963 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:39,963 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:39,971 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:39,973 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,973 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,973 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,973 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,975 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:39,976 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,976 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,976 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,976 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,977 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,982 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,982 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:39,999 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:39,999 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:39,999 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:40,015 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676811504468-1676811519949 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:40,035 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676811504468-1676811519949 of table testRestoreSnapshot-1676811504468 completed 2023-02-19 12:58:40,036 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:40,036 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:40,036 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676811504468-1676811519949 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676811504468 at 1676811519963Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } at 1676811519971 (+8 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676811504468-1676811519949 at 1676811519999 (+28 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676811504468-1676811519949 at 1676811519999Writing data manifest for hbase-failsafe-snaptb1-1676811504468-1676811519949 at 1676811520002 (+3 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676811504468-1676811519949 at 1676811520010 (+8 ms)Snapshot hbase-failsafe-snaptb1-1676811504468-1676811519949 of table testRestoreSnapshot-1676811504468 completed at 1676811520035 (+25 ms) 2023-02-19 12:58:40,037 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE 2023-02-19 12:58:40,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676811504468, type=EXCLUSIVE in 77 msec 2023-02-19 12:58:40,213 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:40,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 } is done 2023-02-19 12:58:40,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676811504468-1676811519949 table=testRestoreSnapshot-1676811504468 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:40,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 ) 2023-02-19 12:58:40,230 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676811504468 as table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-19 12:58:40,242 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000004 2023-02-19 12:58:40,243 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000003 2023-02-19 12:58:40,243 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000004 2023-02-19 12:58:40,243 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } 2023-02-19 12:58:40,246 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:40,246 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,252 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,253 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,257 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,260 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,264 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,274 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,277 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,280 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,283 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,287 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,290 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,293 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,295 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a of snapshot=snaptb1-1676811504468 to region=ad438b5c0e12fc08958125c499973180 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,299 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,299 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,299 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,299 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,299 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,299 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,300 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3809b72d50754a1f92510fd72cc59c86 of snapshot=snaptb1-1676811504468 to region=51f07ae8ab83176a1dfeaae89bd6da17 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,300 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3120ab30e9cd410aa1c81515aec2e8ba of snapshot=snaptb1-1676811504468 to region=e7b05a6d17f916f82251644d3eb30cbb table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,301 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 686fa623ebe140c593a3f885472e7cb2 of snapshot=snaptb1-1676811504468 to region=9073362bb9d4cd684678e7eb8e65b9f1 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,301 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 8bdcfc8bf7444dc9ae76249f5c286818 of snapshot=snaptb1-1676811504468 to region=ac89ddd19c2efa260abf3eee99ed5c81 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,301 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 695ee31eaa9a460b80aa2b4fd5fca7dc of snapshot=snaptb1-1676811504468 to region=60f6a18c9ce621af4e18258a265d998f table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,301 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 60d7d801122244c2923fae4f69e56306 of snapshot=snaptb1-1676811504468 to region=f3edc0852f9dc085eb7e7bda3e53af28 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,306 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d1a1507f32ed4c32a17152c28671d734 of snapshot=snaptb1-1676811504468 to region=51f07ae8ab83176a1dfeaae89bd6da17 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,306 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a3f7fdeb36f24f4199df01d17ca229de of snapshot=snaptb1-1676811504468 to region=ac89ddd19c2efa260abf3eee99ed5c81 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,308 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d7a343597ad64954a497481cd3e01d4d of snapshot=snaptb1-1676811504468 to region=9073362bb9d4cd684678e7eb8e65b9f1 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,308 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c47fcfb7a4ef4b7484a84a75b373dfe0 of snapshot=snaptb1-1676811504468 to region=f3edc0852f9dc085eb7e7bda3e53af28 table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,308 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7ed4e7cce591478da79ea8e1ed713797 of snapshot=snaptb1-1676811504468 to region=e7b05a6d17f916f82251644d3eb30cbb table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,308 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f9a520182acc4e20a59ab4fedc12f191 of snapshot=snaptb1-1676811504468 to region=60f6a18c9ce621af4e18258a265d998f table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,317 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:40,318 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811520318"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:40,318 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811520318"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:40,318 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811520318"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:40,318 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811520318"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:40,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811520318"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:40,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811520318"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:40,321 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811520319"}]},"ts":"1676811520319"} 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811520319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811520319"}]},"ts":"1676811520319"} 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811520319"}]},"ts":"1676811520319"} 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811520319"}]},"ts":"1676811520319"} 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811520319"}]},"ts":"1676811520319"} 2023-02-19 12:58:40,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811520319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811520319"}]},"ts":"1676811520319"} 2023-02-19 12:58:40,323 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-19 12:58:40,323 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-19 12:58:40,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:40,323 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676811504468 table=testRestoreSnapshot-1676811504468 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676811504468 completed! 2023-02-19 12:58:40,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 ) in 96 msec 2023-02-19 12:58:40,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-19 12:58:40,483 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676811504468, procId: 147 completed 2023-02-19 12:58:40,483 INFO [Listener at localhost/43399] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:40,484 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676811504468-1676811519949" 2023-02-19 12:58:40,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676811504468-1676811519949 2023-02-19 12:58:40,487 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,487 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-19 12:58:40,490 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676811504468 2023-02-19 12:58:40,490 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811520490"}]},"ts":"1676811520490"} 2023-02-19 12:58:40,492 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:40,495 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:40,495 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:40,497 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:40,497 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:40,497 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:40,497 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:40,497 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:40,498 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:40,498 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:40,498 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:40,498 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:40,498 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:40,498 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:40,499 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-19 12:58:40,520 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-19 12:58:40,648 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:40,648 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:40,648 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:40,648 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:40,648 DEBUG [jenkins-hbase4:34259] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:40,652 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:40,652 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:40,652 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:40,652 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:40,652 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520652"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811520652"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811520652"}]},"ts":"1676811520652"} 2023-02-19 12:58:40,652 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811520652"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811520652"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811520652"}]},"ts":"1676811520652"} 2023-02-19 12:58:40,652 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:40,652 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520652"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811520652"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811520652"}]},"ts":"1676811520652"} 2023-02-19 12:58:40,652 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811520652"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811520652"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811520652"}]},"ts":"1676811520652"} 2023-02-19 12:58:40,652 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520652"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811520652"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811520652"}]},"ts":"1676811520652"} 2023-02-19 12:58:40,654 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=153, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:40,654 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=154, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:40,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=150, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:40,655 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=149, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:40,656 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=152, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:40,656 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:40,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520656"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811520656"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811520656"}]},"ts":"1676811520656"} 2023-02-19 12:58:40,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:40,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-19 12:58:40,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:40,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:40,809 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:40,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,810 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:40,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,810 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:40,810 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:40,810 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:40,810 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:40,810 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:40,811 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:40,811 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,813 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:40,813 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:40,813 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:40,827 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:40,827 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:40,827 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:40,834 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:40,834 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:40,834 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:40,834 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:40,834 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:40,835 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:40,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:40,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:40,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74528733, jitterRate=0.11056466400623322}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:40,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:40,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71803275, jitterRate=0.06995217502117157}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:40,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:40,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=156, masterSystemTime=1676811520806 2023-02-19 12:58:40,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=155, masterSystemTime=1676811520805 2023-02-19 12:58:40,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:40,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:40,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:40,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:40,843 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:40,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,843 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811520843"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811520843"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811520843"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811520843"}]},"ts":"1676811520843"} 2023-02-19 12:58:40,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:40,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:40,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:40,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:40,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:40,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:40,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,844 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:40,844 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520844"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811520844"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811520844"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811520844"}]},"ts":"1676811520844"} 2023-02-19 12:58:40,845 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,845 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,846 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:40,846 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:40,846 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:40,846 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:40,847 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:40,847 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:40,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=154 2023-02-19 12:58:40,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=154, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 191 msec 2023-02-19 12:58:40,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=153 2023-02-19 12:58:40,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=153, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769 in 194 msec 2023-02-19 12:58:40,850 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 354 msec 2023-02-19 12:58:40,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 354 msec 2023-02-19 12:58:40,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:40,857 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:40,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66816032, jitterRate=-0.004363536834716797}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:40,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:40,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=159, masterSystemTime=1676811520807 2023-02-19 12:58:40,858 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:40,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:40,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:40,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:40,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:40,860 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:40,860 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520860"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811520860"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811520860"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811520860"}]},"ts":"1676811520860"} 2023-02-19 12:58:40,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:40,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,863 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,864 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:40,864 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:40,864 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:40,867 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:40,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=152 2023-02-19 12:58:40,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=152, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696 in 207 msec 2023-02-19 12:58:40,867 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:40,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,871 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:40,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 375 msec 2023-02-19 12:58:40,871 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:40,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:40,878 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:40,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:40,879 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60932512, jitterRate=-0.09203481674194336}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:40,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:40,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=160, masterSystemTime=1676811520806 2023-02-19 12:58:40,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73230202, jitterRate=0.09121504426002502}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:40,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:40,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=157, masterSystemTime=1676811520805 2023-02-19 12:58:40,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:40,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:40,883 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:40,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:40,883 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520882"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811520882"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811520882"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811520882"}]},"ts":"1676811520882"} 2023-02-19 12:58:40,883 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:40,883 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:40,883 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811520883"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811520883"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811520883"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811520883"}]},"ts":"1676811520883"} 2023-02-19 12:58:40,886 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:40,886 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:40,887 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-19 12:58:40,887 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737 in 226 msec 2023-02-19 12:58:40,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,888 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=150 2023-02-19 12:58:40,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,888 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=150, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769 in 231 msec 2023-02-19 12:58:40,888 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 392 msec 2023-02-19 12:58:40,890 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 393 msec 2023-02-19 12:58:40,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:40,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68293641, jitterRate=0.017654553055763245}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:40,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:40,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=158, masterSystemTime=1676811520807 2023-02-19 12:58:40,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:40,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:40,893 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:40,894 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811520893"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811520893"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811520893"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811520893"}]},"ts":"1676811520893"} 2023-02-19 12:58:40,896 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=149 2023-02-19 12:58:40,896 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=149, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 240 msec 2023-02-19 12:58:40,898 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=149, resume processing ppid=148 2023-02-19 12:58:40,898 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 401 msec 2023-02-19 12:58:40,899 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811520898"}]},"ts":"1676811520898"} 2023-02-19 12:58:40,900 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:40,901 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676811504468' was successfully enabled. 2023-02-19 12:58:40,904 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676811504468 in 415 msec 2023-02-19 12:58:41,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-19 12:58:41,242 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 148 completed 2023-02-19 12:58:41,242 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(139): Connect 0x2ad94238 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:41,248 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a5b8d75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:41,249 DEBUG [hconnection-0x131d42f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:41,251 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52318, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:41,254 DEBUG [hconnection-0x131d42f3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:41,255 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:41,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,463 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:41,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,571 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c1707f4 to 127.0.0.1:60853 2023-02-19 12:58:41,571 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:41,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,573 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2606f9a8 to 127.0.0.1:60853 2023-02-19 12:58:41,573 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:41,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,640 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:41,643 DEBUG [hconnection-0x131d42f3-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:41,649 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39852, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:41,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,866 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:41,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:41,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,070 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:42,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,255 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:42,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:42,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:43,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:44,324 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,325 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-19 12:58:44,329 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811524329"}]},"ts":"1676811524329"} 2023-02-19 12:58:44,330 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:44,331 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:44,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:44,334 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:44,334 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:44,334 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:44,334 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:44,334 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:44,335 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:44,335 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:44,335 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811524335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811524335"}]},"ts":"1676811524335"} 2023-02-19 12:58:44,335 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811524335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811524335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811524335"}]},"ts":"1676811524335"} 2023-02-19 12:58:44,335 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:44,335 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:44,335 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:44,335 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811524335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811524335"}]},"ts":"1676811524335"} 2023-02-19 12:58:44,335 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811524335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811524335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811524335"}]},"ts":"1676811524335"} 2023-02-19 12:58:44,335 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811524335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811524335"}]},"ts":"1676811524335"} 2023-02-19 12:58:44,336 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=163, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:44,337 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=162, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:44,338 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=164, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:44,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=167, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:44,340 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=165, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:44,340 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:44,341 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:44,341 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811524341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811524341"}]},"ts":"1676811524341"} 2023-02-19 12:58:44,343 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:44,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:44,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:44,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:44,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:44,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:44,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:44,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-19 12:58:44,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:44,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-19 12:58:44,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-19 12:58:44,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:44,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:44,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:44,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:44,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:44,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,505 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:44,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:44,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,505 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524505"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524505"}]},"ts":"1676811524505"} 2023-02-19 12:58:44,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:44,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:44,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:44,510 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:44,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,510 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811524510"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524510"}]},"ts":"1676811524510"} 2023-02-19 12:58:44,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:44,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:44,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:44,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:44,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:44,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,511 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:44,511 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524511"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524511"}]},"ts":"1676811524511"} 2023-02-19 12:58:44,519 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=163 2023-02-19 12:58:44,519 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=163, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769 in 174 msec 2023-02-19 12:58:44,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-19 12:58:44,519 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=162 2023-02-19 12:58:44,519 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=162, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,41307,1676811501696 in 179 msec 2023-02-19 12:58:44,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:44,520 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 187 msec 2023-02-19 12:58:44,520 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=164 2023-02-19 12:58:44,520 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=164, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,39199,1676811501737 in 180 msec 2023-02-19 12:58:44,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-19 12:58:44,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-19 12:58:44,521 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 187 msec 2023-02-19 12:58:44,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:44,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:44,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:44,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 188 msec 2023-02-19 12:58:44,522 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:44,522 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524522"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524522"}]},"ts":"1676811524522"} 2023-02-19 12:58:44,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:44,523 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:44,523 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524523"}]},"ts":"1676811524523"} 2023-02-19 12:58:44,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:44,523 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:44,523 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811524523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524523"}]},"ts":"1676811524523"} 2023-02-19 12:58:44,525 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-19 12:58:44,525 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,45077,1676811501769 in 180 msec 2023-02-19 12:58:44,525 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=165 2023-02-19 12:58:44,525 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=165, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,41307,1676811501696 in 184 msec 2023-02-19 12:58:44,526 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 193 msec 2023-02-19 12:58:44,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=167 2023-02-19 12:58:44,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=167, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 187 msec 2023-02-19 12:58:44,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 193 msec 2023-02-19 12:58:44,527 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=167, resume processing ppid=161 2023-02-19 12:58:44,527 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 194 msec 2023-02-19 12:58:44,528 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811524527"}]},"ts":"1676811524527"} 2023-02-19 12:58:44,528 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:44,530 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:44,532 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 206 msec 2023-02-19 12:58:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-19 12:58:44,580 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 161 completed 2023-02-19 12:58:44,584 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,591 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,593 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-19 12:58:44,600 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:44,600 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:44,600 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:44,600 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:44,600 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:44,600 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:44,602 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits] 2023-02-19 12:58:44,602 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits] 2023-02-19 12:58:44,602 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits] 2023-02-19 12:58:44,602 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits] 2023-02-19 12:58:44,603 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits] 2023-02-19 12:58:44,603 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits] 2023-02-19 12:58:44,606 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:44,606 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:44,607 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:44,607 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:44,607 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:44,607 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:44,608 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:44,608 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:44,608 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:44,609 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:44,609 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:44,609 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:44,612 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/27.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/27.seqid 2023-02-19 12:58:44,612 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/27.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/27.seqid 2023-02-19 12:58:44,613 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:44,613 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/27.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/27.seqid 2023-02-19 12:58:44,613 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/27.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/27.seqid 2023-02-19 12:58:44,613 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/27.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/27.seqid 2023-02-19 12:58:44,613 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:44,614 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:44,614 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/27.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/27.seqid 2023-02-19 12:58:44,614 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:44,614 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:44,614 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:44,614 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676811504468 regions 2023-02-19 12:58:44,615 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180 2023-02-19 12:58:44,615 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf] 2023-02-19 12:58:44,617 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 2023-02-19 12:58:44,618 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:44,619 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:44,621 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 2023-02-19 12:58:44,622 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 2023-02-19 12:58:44,623 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:44,624 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:44,625 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 2023-02-19 12:58:44,626 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 2023-02-19 12:58:44,627 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:44,628 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 2023-02-19 12:58:44,629 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:44,629 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180 2023-02-19 12:58:44,631 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,653 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676811504468 from hbase:meta 2023-02-19 12:58:44,655 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676811504468' descriptor. 2023-02-19 12:58:44,656 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676811504468' from region states. 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811524657"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811524657"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811524657"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811524657"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811524657"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811524657"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,659 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-19 12:58:44,659 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:44,659 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676811504468' as deleted. 2023-02-19 12:58:44,659 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676811524659"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:44,660 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676811504468 state from META 2023-02-19 12:58:44,662 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,663 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 in 77 msec 2023-02-19 12:58:44,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-19 12:58:44,845 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 174 completed 2023-02-19 12:58:44,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 ) 2023-02-19 12:58:44,871 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676811504468 as table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-19 12:58:44,884 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:44,887 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:44,887 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,887 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,890 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,892 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,894 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,895 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,897 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,899 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,901 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,902 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,904 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,906 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,908 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,909 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:44,909 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:44,909 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=51f07ae8ab83176a1dfeaae89bd6da17 as 51f07ae8ab83176a1dfeaae89bd6da17 in snapshot snaptb1-1676811504468 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=ac89ddd19c2efa260abf3eee99ed5c81 as ac89ddd19c2efa260abf3eee99ed5c81 in snapshot snaptb1-1676811504468 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=f3edc0852f9dc085eb7e7bda3e53af28 as f3edc0852f9dc085eb7e7bda3e53af28 in snapshot snaptb1-1676811504468 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=e7b05a6d17f916f82251644d3eb30cbb as e7b05a6d17f916f82251644d3eb30cbb in snapshot snaptb1-1676811504468 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=60f6a18c9ce621af4e18258a265d998f as 60f6a18c9ce621af4e18258a265d998f in snapshot snaptb1-1676811504468 2023-02-19 12:58:44,910 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=9073362bb9d4cd684678e7eb8e65b9f1 as 9073362bb9d4cd684678e7eb8e65b9f1 in snapshot snaptb1-1676811504468 2023-02-19 12:58:44,911 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:44,911 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:44,912 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:44,912 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:44,912 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:44,912 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676811504468', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:44,923 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:44,924 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:44,924 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 60d7d801122244c2923fae4f69e56306 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,924 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 8bdcfc8bf7444dc9ae76249f5c286818 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,926 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:44,926 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 695ee31eaa9a460b80aa2b4fd5fca7dc from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,929 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:44,929 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c47fcfb7a4ef4b7484a84a75b373dfe0 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,929 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 686fa623ebe140c593a3f885472e7cb2 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,929 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a3f7fdeb36f24f4199df01d17ca229de from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,929 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:44,929 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3120ab30e9cd410aa1c81515aec2e8ba from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,930 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f9a520182acc4e20a59ab4fedc12f191 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,932 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:44,932 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,932 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,932 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:44,932 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,932 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:44,933 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:44,933 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d7a343597ad64954a497481cd3e01d4d from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,933 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7ed4e7cce591478da79ea8e1ed713797 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,933 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:44,933 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,933 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,933 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:44,933 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,933 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:44,933 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:44,934 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:44,934 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,934 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,934 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:44,934 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,934 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:44,934 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:44,934 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:44,934 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3809b72d50754a1f92510fd72cc59c86 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:44,936 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,936 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,936 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,936 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:44,936 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:44,937 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d1a1507f32ed4c32a17152c28671d734 from cloned region in snapshot snaptb1-1676811504468 to table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:44,938 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:44,938 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,938 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,938 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:44,939 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,939 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:44,939 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:44,939 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 2023-02-19 12:58:44,939 INFO [PEWorker-2] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676811504468 on table=testRestoreSnapshot-1676811504468 completed! 2023-02-19 12:58:44,943 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524943"}]},"ts":"1676811524943"} 2023-02-19 12:58:44,943 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524943"}]},"ts":"1676811524943"} 2023-02-19 12:58:44,943 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811524943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524943"}]},"ts":"1676811524943"} 2023-02-19 12:58:44,943 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524943"}]},"ts":"1676811524943"} 2023-02-19 12:58:44,944 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811524943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524943"}]},"ts":"1676811524943"} 2023-02-19 12:58:44,944 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811524943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811524943"}]},"ts":"1676811524943"} 2023-02-19 12:58:44,946 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-19 12:58:44,947 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811524946"}]},"ts":"1676811524946"} 2023-02-19 12:58:44,948 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLING in hbase:meta 2023-02-19 12:58:44,958 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:44,958 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:44,958 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:44,958 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:44,958 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:44,959 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN}] 2023-02-19 12:58:44,961 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN 2023-02-19 12:58:44,961 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN 2023-02-19 12:58:44,961 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN 2023-02-19 12:58:44,961 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN 2023-02-19 12:58:44,962 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=false 2023-02-19 12:58:44,962 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=false 2023-02-19 12:58:44,962 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=false 2023-02-19 12:58:44,962 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:44,962 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN 2023-02-19 12:58:44,962 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN 2023-02-19 12:58:44,963 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=false 2023-02-19 12:58:44,963 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:45,112 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:45,115 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:45,115 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:45,115 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:45,115 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:45,115 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:45,116 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811525115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811525115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811525115"}]},"ts":"1676811525115"} 2023-02-19 12:58:45,116 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811525115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811525115"}]},"ts":"1676811525115"} 2023-02-19 12:58:45,116 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811525115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811525115"}]},"ts":"1676811525115"} 2023-02-19 12:58:45,115 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811525115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811525115"}]},"ts":"1676811525115"} 2023-02-19 12:58:45,116 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811525115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811525115"}]},"ts":"1676811525115"} 2023-02-19 12:58:45,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=176, state=RUNNABLE; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:45,120 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=179, state=RUNNABLE; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:45,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=177, state=RUNNABLE; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:45,122 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=178, state=RUNNABLE; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:45,122 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=180, state=RUNNABLE; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:45,122 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:45,123 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811525122"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811525122"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811525122"}]},"ts":"1676811525122"} 2023-02-19 12:58:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-19 12:58:45,124 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=181, state=RUNNABLE; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:45,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:45,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:45,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:45,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:45,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:45,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:45,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:45,275 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:45,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,276 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:45,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,276 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:45,276 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf 2023-02-19 12:58:45,276 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:45,277 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf 2023-02-19 12:58:45,277 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac89ddd19c2efa260abf3eee99ed5c81 columnFamilyName cf 2023-02-19 12:58:45,277 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,277 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9073362bb9d4cd684678e7eb8e65b9f1 columnFamilyName cf 2023-02-19 12:58:45,277 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:45,277 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf 2023-02-19 12:58:45,278 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7b05a6d17f916f82251644d3eb30cbb columnFamilyName cf 2023-02-19 12:58:45,286 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:45,286 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:45,289 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:45,294 DEBUG [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:45,296 INFO [StoreOpener-ac89ddd19c2efa260abf3eee99ed5c81-1] regionserver.HStore(310): Store=ac89ddd19c2efa260abf3eee99ed5c81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:45,297 DEBUG [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:45,297 INFO [StoreOpener-9073362bb9d4cd684678e7eb8e65b9f1-1] regionserver.HStore(310): Store=9073362bb9d4cd684678e7eb8e65b9f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:45,297 DEBUG [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:45,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,297 INFO [StoreOpener-e7b05a6d17f916f82251644d3eb30cbb-1] regionserver.HStore(310): Store=e7b05a6d17f916f82251644d3eb30cbb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:45,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:45,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:45,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:45,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-19 12:58:45,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-19 12:58:45,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-19 12:58:45,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ac89ddd19c2efa260abf3eee99ed5c81; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63532533, jitterRate=-0.05329148471355438}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:45,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:45,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9073362bb9d4cd684678e7eb8e65b9f1; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68807228, jitterRate=0.02530759572982788}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:45,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:45,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7b05a6d17f916f82251644d3eb30cbb; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59642543, jitterRate=-0.11125685274600983}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:45,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:45,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1., pid=183, masterSystemTime=1676811525272 2023-02-19 12:58:45,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81., pid=184, masterSystemTime=1676811525271 2023-02-19 12:58:45,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb., pid=185, masterSystemTime=1676811525274 2023-02-19 12:58:45,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:45,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:45,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:45,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:45,305 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:45,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:45,305 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525305"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811525305"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811525305"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811525305"}]},"ts":"1676811525305"} 2023-02-19 12:58:45,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:45,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:45,306 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:45,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:45,306 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525306"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811525306"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811525306"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811525306"}]},"ts":"1676811525306"} 2023-02-19 12:58:45,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:45,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:45,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,307 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:45,307 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:45,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:45,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:45,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:45,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676811504468 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:45,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,307 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525307"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811525307"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811525307"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811525307"}]},"ts":"1676811525307"} 2023-02-19 12:58:45,308 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:45,308 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,308 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf 2023-02-19 12:58:45,308 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,309 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3edc0852f9dc085eb7e7bda3e53af28 columnFamilyName cf 2023-02-19 12:58:45,310 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:45,310 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf 2023-02-19 12:58:45,310 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:45,310 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf 2023-02-19 12:58:45,310 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51f07ae8ab83176a1dfeaae89bd6da17 columnFamilyName cf 2023-02-19 12:58:45,310 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 60f6a18c9ce621af4e18258a265d998f columnFamilyName cf 2023-02-19 12:58:45,314 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=178 2023-02-19 12:58:45,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=179 2023-02-19 12:58:45,314 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=178, state=SUCCESS; OpenRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,41307,1676811501696 in 186 msec 2023-02-19 12:58:45,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=179, state=SUCCESS; OpenRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 189 msec 2023-02-19 12:58:45,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=177 2023-02-19 12:58:45,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=177, state=SUCCESS; OpenRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769 in 188 msec 2023-02-19 12:58:45,316 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, ASSIGN in 355 msec 2023-02-19 12:58:45,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, ASSIGN in 355 msec 2023-02-19 12:58:45,319 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, ASSIGN in 357 msec 2023-02-19 12:58:45,320 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:45,320 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:45,321 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:45,324 DEBUG [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:45,324 INFO [StoreOpener-51f07ae8ab83176a1dfeaae89bd6da17-1] regionserver.HStore(310): Store=51f07ae8ab83176a1dfeaae89bd6da17/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:45,324 DEBUG [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:45,325 INFO [StoreOpener-f3edc0852f9dc085eb7e7bda3e53af28-1] regionserver.HStore(310): Store=f3edc0852f9dc085eb7e7bda3e53af28/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:45,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,326 DEBUG [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:45,326 INFO [StoreOpener-60f6a18c9ce621af4e18258a265d998f-1] regionserver.HStore(310): Store=60f6a18c9ce621af4e18258a265d998f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:45,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:45,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:45,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:45,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-19 12:58:45,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-19 12:58:45,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 51f07ae8ab83176a1dfeaae89bd6da17; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67945690, jitterRate=0.012469679117202759}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:45,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:45,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3edc0852f9dc085eb7e7bda3e53af28; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60869516, jitterRate=-0.09297353029251099}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:45,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:45,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17., pid=186, masterSystemTime=1676811525274 2023-02-19 12:58:45,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28., pid=187, masterSystemTime=1676811525272 2023-02-19 12:58:45,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:45,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:45,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-19 12:58:45,333 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:45,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 60f6a18c9ce621af4e18258a265d998f; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59476991, jitterRate=-0.1137237697839737}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:45,333 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811525333"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811525333"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811525333"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811525333"}]},"ts":"1676811525333"} 2023-02-19 12:58:45,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:45,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:45,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:45,334 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:45,334 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811525334"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811525334"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811525334"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811525334"}]},"ts":"1676811525334"} 2023-02-19 12:58:45,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f., pid=182, masterSystemTime=1676811525271 2023-02-19 12:58:45,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:45,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:45,336 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:45,336 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811525336"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811525336"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811525336"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811525336"}]},"ts":"1676811525336"} 2023-02-19 12:58:45,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=180 2023-02-19 12:58:45,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=180, state=SUCCESS; OpenRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 213 msec 2023-02-19 12:58:45,337 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=181 2023-02-19 12:58:45,337 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=181, state=SUCCESS; OpenRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 212 msec 2023-02-19 12:58:45,338 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, ASSIGN in 378 msec 2023-02-19 12:58:45,338 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, ASSIGN in 378 msec 2023-02-19 12:58:45,339 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=176 2023-02-19 12:58:45,339 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=176, state=SUCCESS; OpenRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,45077,1676811501769 in 220 msec 2023-02-19 12:58:45,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=176, resume processing ppid=175 2023-02-19 12:58:45,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, ASSIGN in 380 msec 2023-02-19 12:58:45,341 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811525341"}]},"ts":"1676811525341"} 2023-02-19 12:58:45,342 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=ENABLED in hbase:meta 2023-02-19 12:58:45,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676811504468 snapshot=name: "snaptb1-1676811504468" table: "testRestoreSnapshot-1676811504468" creation_time: 1676811515134 type: DISABLED version: 2 ttl: 0 ) in 480 msec 2023-02-19 12:58:45,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-19 12:58:45,626 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676811504468, procId: 175 completed 2023-02-19 12:58:45,626 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(139): Connect 0x3e1138f2 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:45,634 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@594888d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:45,636 DEBUG [hconnection-0x7254d96e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:45,639 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39588, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:45,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,827 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,947 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:45,950 DEBUG [hconnection-0x7254d96e-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:45,951 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:45,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,173 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:46,176 DEBUG [hconnection-0x7254d96e-shared-pool-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:46,177 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50136, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:46,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,311 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-19 12:58:46,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,367 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:46,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,514 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:46,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:46,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,332 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ad94238 to 127.0.0.1:60853 2023-02-19 12:58:47,332 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:47,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:47,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:48,327 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,327 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-19 12:58:48,331 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811528331"}]},"ts":"1676811528331"} 2023-02-19 12:58:48,332 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLING in hbase:meta 2023-02-19 12:58:48,334 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676811504468 to state=DISABLING 2023-02-19 12:58:48,334 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN}] 2023-02-19 12:58:48,336 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN 2023-02-19 12:58:48,336 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN 2023-02-19 12:58:48,336 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN 2023-02-19 12:58:48,336 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN 2023-02-19 12:58:48,336 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN 2023-02-19 12:58:48,337 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:48,337 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:48,337 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:48,337 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528337"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811528337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811528337"}]},"ts":"1676811528337"} 2023-02-19 12:58:48,337 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528337"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811528337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811528337"}]},"ts":"1676811528337"} 2023-02-19 12:58:48,337 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:48,337 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:48,337 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811528337"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811528337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811528337"}]},"ts":"1676811528337"} 2023-02-19 12:58:48,337 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528337"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811528337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811528337"}]},"ts":"1676811528337"} 2023-02-19 12:58:48,337 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811528337"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811528337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811528337"}]},"ts":"1676811528337"} 2023-02-19 12:58:48,339 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=191, state=RUNNABLE; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:48,339 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=193, state=RUNNABLE; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:48,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=189, state=RUNNABLE; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:48,343 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=190, state=RUNNABLE; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:48,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=194, state=RUNNABLE; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:48,344 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN 2023-02-19 12:58:48,344 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:48,344 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528344"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811528344"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811528344"}]},"ts":"1676811528344"} 2023-02-19 12:58:48,346 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=192, state=RUNNABLE; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:48,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:48,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7b05a6d17f916f82251644d3eb30cbb, disabling compactions & flushes 2023-02-19 12:58:48,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:48,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:48,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. after waiting 0 ms 2023-02-19 12:58:48,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:48,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:48,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:48,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9073362bb9d4cd684678e7eb8e65b9f1, disabling compactions & flushes 2023-02-19 12:58:48,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 60f6a18c9ce621af4e18258a265d998f, disabling compactions & flushes 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:48,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. after waiting 0 ms 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. after waiting 0 ms 2023-02-19 12:58:48,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:48,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-19 12:58:48,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb. 2023-02-19 12:58:48,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7b05a6d17f916f82251644d3eb30cbb: 2023-02-19 12:58:48,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-19 12:58:48,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f. 2023-02-19 12:58:48,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 60f6a18c9ce621af4e18258a265d998f: 2023-02-19 12:58:48,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-19 12:58:48,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:48,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1. 2023-02-19 12:58:48,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:48,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9073362bb9d4cd684678e7eb8e65b9f1: 2023-02-19 12:58:48,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 51f07ae8ab83176a1dfeaae89bd6da17, disabling compactions & flushes 2023-02-19 12:58:48,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:48,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:48,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. after waiting 0 ms 2023-02-19 12:58:48,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:48,509 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=e7b05a6d17f916f82251644d3eb30cbb, regionState=CLOSED 2023-02-19 12:58:48,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528509"}]},"ts":"1676811528509"} 2023-02-19 12:58:48,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:48,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:48,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ac89ddd19c2efa260abf3eee99ed5c81, disabling compactions & flushes 2023-02-19 12:58:48,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:48,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:48,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. after waiting 0 ms 2023-02-19 12:58:48,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:48,517 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=60f6a18c9ce621af4e18258a265d998f, regionState=CLOSED 2023-02-19 12:58:48,517 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811528517"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528517"}]},"ts":"1676811528517"} 2023-02-19 12:58:48,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:48,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:48,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3edc0852f9dc085eb7e7bda3e53af28, disabling compactions & flushes 2023-02-19 12:58:48,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:48,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:48,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. after waiting 0 ms 2023-02-19 12:58:48,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:48,518 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=9073362bb9d4cd684678e7eb8e65b9f1, regionState=CLOSED 2023-02-19 12:58:48,518 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528518"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528518"}]},"ts":"1676811528518"} 2023-02-19 12:58:48,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=191 2023-02-19 12:58:48,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=191, state=SUCCESS; CloseRegionProcedure e7b05a6d17f916f82251644d3eb30cbb, server=jenkins-hbase4.apache.org,41307,1676811501696 in 180 msec 2023-02-19 12:58:48,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=189 2023-02-19 12:58:48,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=189, state=SUCCESS; CloseRegionProcedure 60f6a18c9ce621af4e18258a265d998f, server=jenkins-hbase4.apache.org,45077,1676811501769 in 179 msec 2023-02-19 12:58:48,522 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=e7b05a6d17f916f82251644d3eb30cbb, UNASSIGN in 187 msec 2023-02-19 12:58:48,522 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=192 2023-02-19 12:58:48,523 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=192, state=SUCCESS; CloseRegionProcedure 9073362bb9d4cd684678e7eb8e65b9f1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 174 msec 2023-02-19 12:58:48,523 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=60f6a18c9ce621af4e18258a265d998f, UNASSIGN in 187 msec 2023-02-19 12:58:48,523 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=9073362bb9d4cd684678e7eb8e65b9f1, UNASSIGN in 189 msec 2023-02-19 12:58:48,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-19 12:58:48,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-19 12:58:48,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17. 2023-02-19 12:58:48,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 51f07ae8ab83176a1dfeaae89bd6da17: 2023-02-19 12:58:48,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81. 2023-02-19 12:58:48,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ac89ddd19c2efa260abf3eee99ed5c81: 2023-02-19 12:58:48,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:48,530 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=51f07ae8ab83176a1dfeaae89bd6da17, regionState=CLOSED 2023-02-19 12:58:48,530 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528530"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528530"}]},"ts":"1676811528530"} 2023-02-19 12:58:48,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:48,530 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=ac89ddd19c2efa260abf3eee99ed5c81, regionState=CLOSED 2023-02-19 12:58:48,530 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676811528530"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528530"}]},"ts":"1676811528530"} 2023-02-19 12:58:48,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-19 12:58:48,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28. 2023-02-19 12:58:48,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3edc0852f9dc085eb7e7bda3e53af28: 2023-02-19 12:58:48,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:48,534 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=f3edc0852f9dc085eb7e7bda3e53af28, regionState=CLOSED 2023-02-19 12:58:48,534 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676811528534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528534"}]},"ts":"1676811528534"} 2023-02-19 12:58:48,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=193 2023-02-19 12:58:48,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=193, state=SUCCESS; CloseRegionProcedure 51f07ae8ab83176a1dfeaae89bd6da17, server=jenkins-hbase4.apache.org,41307,1676811501696 in 193 msec 2023-02-19 12:58:48,535 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=190 2023-02-19 12:58:48,535 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=190, state=SUCCESS; CloseRegionProcedure ac89ddd19c2efa260abf3eee99ed5c81, server=jenkins-hbase4.apache.org,45077,1676811501769 in 192 msec 2023-02-19 12:58:48,535 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=51f07ae8ab83176a1dfeaae89bd6da17, UNASSIGN in 200 msec 2023-02-19 12:58:48,536 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=ac89ddd19c2efa260abf3eee99ed5c81, UNASSIGN in 201 msec 2023-02-19 12:58:48,536 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=194 2023-02-19 12:58:48,536 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=194, state=SUCCESS; CloseRegionProcedure f3edc0852f9dc085eb7e7bda3e53af28, server=jenkins-hbase4.apache.org,39199,1676811501737 in 191 msec 2023-02-19 12:58:48,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=194, resume processing ppid=188 2023-02-19 12:58:48,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676811504468, region=f3edc0852f9dc085eb7e7bda3e53af28, UNASSIGN in 202 msec 2023-02-19 12:58:48,538 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811528538"}]},"ts":"1676811528538"} 2023-02-19 12:58:48,539 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676811504468, state=DISABLED in hbase:meta 2023-02-19 12:58:48,540 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676811504468 to state=DISABLED 2023-02-19 12:58:48,542 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676811504468 in 214 msec 2023-02-19 12:58:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-19 12:58:48,582 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 188 completed 2023-02-19 12:58:48,582 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,585 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,585 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-19 12:58:48,592 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:48,592 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:48,592 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:48,592 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:48,592 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:48,592 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:48,594 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits] 2023-02-19 12:58:48,594 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits] 2023-02-19 12:58:48,594 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits] 2023-02-19 12:58:48,594 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits] 2023-02-19 12:58:48,594 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits] 2023-02-19 12:58:48,595 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits] 2023-02-19 12:58:48,598 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524605 currentMtime: 1676811524936 2023-02-19 12:58:48,598 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524605 currentMtime: 1676811524930 2023-02-19 12:58:48,599 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524606 currentMtime: 1676811524932 2023-02-19 12:58:48,599 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524606 currentMtime: 1676811524928 2023-02-19 12:58:48,599 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86.1676811528594. 2023-02-19 12:58:48,599 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524605 currentMtime: 1676811524933 2023-02-19 12:58:48,599 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2.1676811528594. 2023-02-19 12:58:48,599 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc.1676811528594. 2023-02-19 12:58:48,599 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818.1676811528594. 2023-02-19 12:58:48,600 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba.1676811528594. 2023-02-19 12:58:48,600 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524605 currentMtime: 1676811524928 2023-02-19 12:58:48,600 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-686fa623ebe140c593a3f885472e7cb2 2023-02-19 12:58:48,601 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-3809b72d50754a1f92510fd72cc59c86 2023-02-19 12:58:48,601 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-8bdcfc8bf7444dc9ae76249f5c286818 2023-02-19 12:58:48,601 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-695ee31eaa9a460b80aa2b4fd5fca7dc 2023-02-19 12:58:48,601 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306.1676811528595. 2023-02-19 12:58:48,601 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-3120ab30e9cd410aa1c81515aec2e8ba 2023-02-19 12:58:48,602 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-60d7d801122244c2923fae4f69e56306 2023-02-19 12:58:48,602 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524608 currentMtime: 1676811524935 2023-02-19 12:58:48,603 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524608 currentMtime: 1676811524932 2023-02-19 12:58:48,603 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524607 currentMtime: 1676811524938 2023-02-19 12:58:48,603 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524607 currentMtime: 1676811524935 2023-02-19 12:58:48,603 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734.1676811528594. 2023-02-19 12:58:48,603 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524607 currentMtime: 1676811524933 2023-02-19 12:58:48,604 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d.1676811528594. 2023-02-19 12:58:48,604 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de.1676811528594. 2023-02-19 12:58:48,604 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797.1676811528594. 2023-02-19 12:58:48,604 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524607 currentMtime: 1676811524933 2023-02-19 12:58:48,604 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191.1676811528594. 2023-02-19 12:58:48,605 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/cf/testRestoreSnapshot-1676811504468=51f07ae8ab83176a1dfeaae89bd6da17-d1a1507f32ed4c32a17152c28671d734 2023-02-19 12:58:48,605 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/cf/testRestoreSnapshot-1676811504468=9073362bb9d4cd684678e7eb8e65b9f1-d7a343597ad64954a497481cd3e01d4d 2023-02-19 12:58:48,605 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0.1676811528595. 2023-02-19 12:58:48,606 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/cf/testRestoreSnapshot-1676811504468=e7b05a6d17f916f82251644d3eb30cbb-7ed4e7cce591478da79ea8e1ed713797 2023-02-19 12:58:48,606 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/cf/testRestoreSnapshot-1676811504468=ac89ddd19c2efa260abf3eee99ed5c81-a3f7fdeb36f24f4199df01d17ca229de 2023-02-19 12:58:48,606 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/cf/testRestoreSnapshot-1676811504468=60f6a18c9ce621af4e18258a265d998f-f9a520182acc4e20a59ab4fedc12f191 2023-02-19 12:58:48,607 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/cf/testRestoreSnapshot-1676811504468=f3edc0852f9dc085eb7e7bda3e53af28-c47fcfb7a4ef4b7484a84a75b373dfe0 2023-02-19 12:58:48,609 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1/recovered.edits/18.seqid 2023-02-19 12:58:48,609 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17/recovered.edits/18.seqid 2023-02-19 12:58:48,610 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/9073362bb9d4cd684678e7eb8e65b9f1 2023-02-19 12:58:48,610 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f/recovered.edits/18.seqid 2023-02-19 12:58:48,610 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb/recovered.edits/18.seqid 2023-02-19 12:58:48,610 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/51f07ae8ab83176a1dfeaae89bd6da17 2023-02-19 12:58:48,610 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81/recovered.edits/18.seqid 2023-02-19 12:58:48,611 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/60f6a18c9ce621af4e18258a265d998f 2023-02-19 12:58:48,611 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/e7b05a6d17f916f82251644d3eb30cbb 2023-02-19 12:58:48,611 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28/recovered.edits/18.seqid 2023-02-19 12:58:48,611 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/ac89ddd19c2efa260abf3eee99ed5c81 2023-02-19 12:58:48,611 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testRestoreSnapshot-1676811504468/f3edc0852f9dc085eb7e7bda3e53af28 2023-02-19 12:58:48,612 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676811504468 regions 2023-02-19 12:58:48,612 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180 2023-02-19 12:58:48,613 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf] 2023-02-19 12:58:48,615 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524617 currentMtime: 1676811524889 2023-02-19 12:58:48,615 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4.1676811528613. 2023-02-19 12:58:48,616 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c202302199d24a249bf51499e890f0f6e35de3cc4 2023-02-19 12:58:48,617 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524618 currentMtime: 1676811524891 2023-02-19 12:58:48,618 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b.1676811528613. 2023-02-19 12:58:48,619 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-a87ff679a2f3e71d9181a67b7542122c20230219b28b9b4112544e5eb33b18857b75389b 2023-02-19 12:58:48,620 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524619 currentMtime: 1676811524893 2023-02-19 12:58:48,621 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee.1676811528613. 2023-02-19 12:58:48,622 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b202302196fd6c3131dbf4b2080011704fef5dbee 2023-02-19 12:58:48,622 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524620 currentMtime: 1676811524895 2023-02-19 12:58:48,623 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61.1676811528613. 2023-02-19 12:58:48,624 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c4ca4238a0b923820dcc509a6f75849b20230219ed2ee0591da64b548e5e44494fdd7b61 2023-02-19 12:58:48,625 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524621 currentMtime: 1676811524897 2023-02-19 12:58:48,625 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7.1676811528613. 2023-02-19 12:58:48,626 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219cb6057c3b971430dbaa89b096b63aba7 2023-02-19 12:58:48,627 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524622 currentMtime: 1676811524899 2023-02-19 12:58:48,627 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0.1676811528613. 2023-02-19 12:58:48,628 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-c81e728d9d4c2f636f067f89cc14862c20230219ee64c4e6ddbe442c93e9dd0bce974ae0 2023-02-19 12:58:48,630 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524623 currentMtime: 1676811524901 2023-02-19 12:58:48,630 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29.1676811528613. 2023-02-19 12:58:48,631 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e202302190ca6fe6c82974e3cb6412369a4f81a29 2023-02-19 12:58:48,632 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524624 currentMtime: 1676811524902 2023-02-19 12:58:48,632 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8.1676811528613. 2023-02-19 12:58:48,633 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-d41d8cd98f00b204e9800998ecf8427e20230219dedbce397bdb49f0b9a83d61e9a8bcf8 2023-02-19 12:58:48,634 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524625 currentMtime: 1676811524904 2023-02-19 12:58:48,634 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39.1676811528613. 2023-02-19 12:58:48,635 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302190fffae971d48454a8f1f7dabba9e9c39 2023-02-19 12:58:48,636 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524626 currentMtime: 1676811524906 2023-02-19 12:58:48,636 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022.1676811528613. 2023-02-19 12:58:48,637 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-e4da3b7fbbce2345d7772b0674a318d5202302198d322997e99f4d0ea17e10477d994022 2023-02-19 12:58:48,638 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524627 currentMtime: 1676811524907 2023-02-19 12:58:48,638 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254.1676811528613. 2023-02-19 12:58:48,639 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219393d31f899fe46ebbc665e7d0b081254 2023-02-19 12:58:48,640 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a. archiveLen: 0 currentLen: 0 archiveMtime: 1676811524628 currentMtime: 1676811524909 2023-02-19 12:58:48,640 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a.1676811528613. 2023-02-19 12:58:48,641 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180/cf/testRestoreSnapshot-1676811504468=ad438b5c0e12fc08958125c499973180-eccbc87e4b5ce2fe28308fd9f2a7baf320230219854a190928c04a6395607caeeaa2be5a 2023-02-19 12:58:48,642 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testRestoreSnapshot-1676811504468/ad438b5c0e12fc08958125c499973180 2023-02-19 12:58:48,643 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,646 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676811504468 from hbase:meta 2023-02-19 12:58:48,648 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676811504468' descriptor. 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676811504468' from region states. 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811528649"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811528649"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811528649"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811528649"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811528649"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,649 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811528649"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,651 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-19 12:58:48,651 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 60f6a18c9ce621af4e18258a265d998f, NAME => 'testRestoreSnapshot-1676811504468,,1676811504491.60f6a18c9ce621af4e18258a265d998f.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ac89ddd19c2efa260abf3eee99ed5c81, NAME => 'testRestoreSnapshot-1676811504468,1,1676811504491.ac89ddd19c2efa260abf3eee99ed5c81.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e7b05a6d17f916f82251644d3eb30cbb, NAME => 'testRestoreSnapshot-1676811504468,2,1676811504491.e7b05a6d17f916f82251644d3eb30cbb.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 9073362bb9d4cd684678e7eb8e65b9f1, NAME => 'testRestoreSnapshot-1676811504468,3,1676811504491.9073362bb9d4cd684678e7eb8e65b9f1.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 51f07ae8ab83176a1dfeaae89bd6da17, NAME => 'testRestoreSnapshot-1676811504468,4,1676811504491.51f07ae8ab83176a1dfeaae89bd6da17.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => f3edc0852f9dc085eb7e7bda3e53af28, NAME => 'testRestoreSnapshot-1676811504468,5,1676811504491.f3edc0852f9dc085eb7e7bda3e53af28.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:48,651 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676811504468' as deleted. 2023-02-19 12:58:48,651 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676811504468","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676811528651"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:48,653 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676811504468 state from META 2023-02-19 12:58:48,654 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 2023-02-19 12:58:48,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676811504468 in 73 msec 2023-02-19 12:58:48,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-19 12:58:48,837 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676811504468, procId: 201 completed 2023-02-19 12:58:48,842 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676811504468" 2023-02-19 12:58:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676811504468 2023-02-19 12:58:48,845 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676811504468" 2023-02-19 12:58:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676811504468 2023-02-19 12:58:48,847 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676811504468" 2023-02-19 12:58:48,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676811504468 2023-02-19 12:58:48,875 INFO [Listener at localhost/43399] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=425 (was 359) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x514281c-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1572513154_15 at /127.0.0.1:46064 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_630443903_15 at /127.0.0.1:59122 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_630443903_15 at /127.0.0.1:59136 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1853038609_15 at /127.0.0.1:59162 [Waiting for operation #506] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60853@0x3e1138f2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/39761898.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1853038609_15 at /127.0.0.1:45720 [Waiting for operation #527] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1572513154_15 at /127.0.0.1:59160 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x514281c-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x514281c-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1853038609_15 at /127.0.0.1:46052 [Waiting for operation #539] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1572513154_15 at /127.0.0.1:45316 [Waiting for operation #544] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60853@0x3e1138f2-SendThread(127.0.0.1:60853) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1572513154_15 at /127.0.0.1:46074 [Waiting for operation #500] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2128308608_15 at /127.0.0.1:59130 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2128308608_15 at /127.0.0.1:47196 [Waiting for operation #490] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1853038609_15 at /127.0.0.1:45812 [Waiting for operation #786] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x7254d96e-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7254d96e-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x514281c-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1853038609_15 at /127.0.0.1:53554 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1853038609_15 at /127.0.0.1:53576 [Waiting for operation #479] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60853@0x3e1138f2-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x7254d96e-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2128308608_15 at /127.0.0.1:53542 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_630443903_15 at /127.0.0.1:59146 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=805 (was 695) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=109 (was 137), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=9381 (was 9877) 2023-02-19 12:58:48,891 INFO [Listener at localhost/43399] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=425, OpenFileDescriptor=805, MaxFileDescriptor=60000, SystemLoadAverage=109, ProcessCount=170, AvailableMemoryMB=9380 2023-02-19 12:58:48,893 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-19 12:58:48,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:48,895 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-19 12:58:48,896 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676811528891" procId is: 202 2023-02-19 12:58:48,896 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-19 12:58:48,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-19 12:58:48,900 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:48,900 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:48,900 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:48,900 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:48,900 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:48,900 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e empty. 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 empty. 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 empty. 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 empty. 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 empty. 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce empty. 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:48,901 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:48,902 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:48,902 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:48,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676811528891 regions 2023-02-19 12:58:48,916 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/.tabledesc/.tableinfo.0000000001 2023-02-19 12:58:48,917 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9c525e8dcdbb974e61fda246b8d37872, NAME => 'testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:48,918 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(7671): creating {ENCODED => 92751fca21674e6b7ab33630e2b6a119, NAME => 'testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:48,918 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(7671): creating {ENCODED => c6d304ba41abeb8e6ae55cbd4c42f7c1, NAME => 'testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:48,936 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:48,936 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1603): Closing 92751fca21674e6b7ab33630e2b6a119, disabling compactions & flushes 2023-02-19 12:58:48,937 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1603): Closing c6d304ba41abeb8e6ae55cbd4c42f7c1, disabling compactions & flushes 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. after waiting 0 ms 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:48,937 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:48,937 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. after waiting 0 ms 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1557): Region close journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:48,937 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:48,937 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1557): Region close journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:48,937 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(7671): creating {ENCODED => 9afa96a011d8f82b0bebbd3bac2f1fce, NAME => 'testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:48,938 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(7671): creating {ENCODED => 4aca2f2bf5da4c990f8539f674ac2e3e, NAME => 'testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:48,938 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:48,938 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1603): Closing 9c525e8dcdbb974e61fda246b8d37872, disabling compactions & flushes 2023-02-19 12:58:48,938 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:48,938 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:48,938 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. after waiting 0 ms 2023-02-19 12:58:48,938 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:48,938 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:48,938 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1557): Region close journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:48,939 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(7671): creating {ENCODED => 0056208a08192d81a1d3f47ace8488e5, NAME => 'testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676811528891', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp 2023-02-19 12:58:48,946 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:48,946 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1603): Closing 9afa96a011d8f82b0bebbd3bac2f1fce, disabling compactions & flushes 2023-02-19 12:58:48,946 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:48,946 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:48,946 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. after waiting 0 ms 2023-02-19 12:58:48,947 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:48,947 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:48,947 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-2] regionserver.HRegion(1557): Region close journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:48,949 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:48,949 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1603): Closing 4aca2f2bf5da4c990f8539f674ac2e3e, disabling compactions & flushes 2023-02-19 12:58:48,949 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:48,950 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:48,950 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. after waiting 0 ms 2023-02-19 12:58:48,950 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:48,950 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:48,950 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-1] regionserver.HRegion(1557): Region close journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:48,951 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:48,951 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1603): Closing 0056208a08192d81a1d3f47ace8488e5, disabling compactions & flushes 2023-02-19 12:58:48,951 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:48,951 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:48,951 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. after waiting 0 ms 2023-02-19 12:58:48,951 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:48,951 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:48,951 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676811528891-pool-0] regionserver.HRegion(1557): Region close journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:48,953 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 execute state=CREATE_TABLE_ADD_TO_META 2023-02-19 12:58:48,954 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811528954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528954"}]},"ts":"1676811528954"} 2023-02-19 12:58:48,954 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811528954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528954"}]},"ts":"1676811528954"} 2023-02-19 12:58:48,954 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811528954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528954"}]},"ts":"1676811528954"} 2023-02-19 12:58:48,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811528954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528954"}]},"ts":"1676811528954"} 2023-02-19 12:58:48,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811528954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528954"}]},"ts":"1676811528954"} 2023-02-19 12:58:48,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811528954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811528954"}]},"ts":"1676811528954"} 2023-02-19 12:58:48,959 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-19 12:58:48,960 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-19 12:58:48,960 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811528960"}]},"ts":"1676811528960"} 2023-02-19 12:58:48,961 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=ENABLING in hbase:meta 2023-02-19 12:58:48,966 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-19 12:58:48,968 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-19 12:58:48,968 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-19 12:58:48,968 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-19 12:58:48,968 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-19 12:58:48,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN}] 2023-02-19 12:58:48,970 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN 2023-02-19 12:58:48,970 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN 2023-02-19 12:58:48,970 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN 2023-02-19 12:58:48,970 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN 2023-02-19 12:58:48,971 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=false 2023-02-19 12:58:48,971 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=false 2023-02-19 12:58:48,971 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:48,971 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=false 2023-02-19 12:58:48,971 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN 2023-02-19 12:58:48,971 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN 2023-02-19 12:58:48,972 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=false 2023-02-19 12:58:48,972 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=false 2023-02-19 12:58:49,121 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:49,124 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:49,124 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:49,124 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529124"}]},"ts":"1676811529124"} 2023-02-19 12:58:49,124 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:49,124 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:49,125 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529124"}]},"ts":"1676811529124"} 2023-02-19 12:58:49,124 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:49,125 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529124"}]},"ts":"1676811529124"} 2023-02-19 12:58:49,125 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529124"}]},"ts":"1676811529124"} 2023-02-19 12:58:49,125 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529124"}]},"ts":"1676811529124"} 2023-02-19 12:58:49,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=203, state=RUNNABLE; OpenRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:49,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=204, state=RUNNABLE; OpenRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:49,127 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=206, state=RUNNABLE; OpenRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:49,128 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:49,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=208, state=RUNNABLE; OpenRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:49,129 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:49,129 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529129"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529129"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529129"}]},"ts":"1676811529129"} 2023-02-19 12:58:49,130 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=207, state=RUNNABLE; OpenRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:49,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-19 12:58:49,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0056208a08192d81a1d3f47ace8488e5, NAME => 'testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:49,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4aca2f2bf5da4c990f8539f674ac2e3e, NAME => 'testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:49,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 92751fca21674e6b7ab33630e2b6a119, NAME => 'testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,282 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,282 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,282 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,283 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf 2023-02-19 12:58:49,283 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf 2023-02-19 12:58:49,283 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf 2023-02-19 12:58:49,283 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf 2023-02-19 12:58:49,284 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf 2023-02-19 12:58:49,283 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf 2023-02-19 12:58:49,284 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92751fca21674e6b7ab33630e2b6a119 columnFamilyName cf 2023-02-19 12:58:49,284 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0056208a08192d81a1d3f47ace8488e5 columnFamilyName cf 2023-02-19 12:58:49,284 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4aca2f2bf5da4c990f8539f674ac2e3e columnFamilyName cf 2023-02-19 12:58:49,285 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(310): Store=92751fca21674e6b7ab33630e2b6a119/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:49,285 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(310): Store=0056208a08192d81a1d3f47ace8488e5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:49,285 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(310): Store=4aca2f2bf5da4c990f8539f674ac2e3e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:49,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:49,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:49,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:49,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 92751fca21674e6b7ab33630e2b6a119; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75210516, jitterRate=0.12072402238845825}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:49,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4aca2f2bf5da4c990f8539f674ac2e3e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73254319, jitterRate=0.09157441556453705}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:49,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:49,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0056208a08192d81a1d3f47ace8488e5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66675135, jitterRate=-0.00646306574344635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:49,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:49,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:49,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e., pid=214, masterSystemTime=1676811529278 2023-02-19 12:58:49,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5., pid=213, masterSystemTime=1676811529278 2023-02-19 12:58:49,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119., pid=212, masterSystemTime=1676811529279 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c6d304ba41abeb8e6ae55cbd4c42f7c1, NAME => 'testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:49,294 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,294 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529294"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811529294"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811529294"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811529294"}]},"ts":"1676811529294"} 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9c525e8dcdbb974e61fda246b8d37872, NAME => 'testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:49,295 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,295 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529295"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811529295"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811529295"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811529295"}]},"ts":"1676811529295"} 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9afa96a011d8f82b0bebbd3bac2f1fce, NAME => 'testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:49,296 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,296 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:49,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,296 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529296"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811529296"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811529296"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811529296"}]},"ts":"1676811529296"} 2023-02-19 12:58:49,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:49,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,297 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,297 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=207 2023-02-19 12:58:49,297 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,298 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf 2023-02-19 12:58:49,297 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=207, state=SUCCESS; OpenRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737 in 166 msec 2023-02-19 12:58:49,298 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf 2023-02-19 12:58:49,298 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6d304ba41abeb8e6ae55cbd4c42f7c1 columnFamilyName cf 2023-02-19 12:58:49,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=208 2023-02-19 12:58:49,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=208, state=SUCCESS; OpenRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696 in 167 msec 2023-02-19 12:58:49,299 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf 2023-02-19 12:58:49,299 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf 2023-02-19 12:58:49,299 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf 2023-02-19 12:58:49,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN in 330 msec 2023-02-19 12:58:49,299 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf 2023-02-19 12:58:49,299 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(310): Store=c6d304ba41abeb8e6ae55cbd4c42f7c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:49,299 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9afa96a011d8f82b0bebbd3bac2f1fce columnFamilyName cf 2023-02-19 12:58:49,300 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9c525e8dcdbb974e61fda246b8d37872 columnFamilyName cf 2023-02-19 12:58:49,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-19 12:58:49,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769 in 170 msec 2023-02-19 12:58:49,300 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN in 331 msec 2023-02-19 12:58:49,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,300 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(310): Store=9afa96a011d8f82b0bebbd3bac2f1fce/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:49,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,302 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(310): Store=9c525e8dcdbb974e61fda246b8d37872/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:49,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,303 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN in 332 msec 2023-02-19 12:58:49,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:49,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c6d304ba41abeb8e6ae55cbd4c42f7c1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66432546, jitterRate=-0.01007792353630066}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:49,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9afa96a011d8f82b0bebbd3bac2f1fce; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65381389, jitterRate=-0.02574138343334198}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:49,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:49,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1., pid=210, masterSystemTime=1676811529278 2023-02-19 12:58:49,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce., pid=211, masterSystemTime=1676811529279 2023-02-19 12:58:49,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-19 12:58:49,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9c525e8dcdbb974e61fda246b8d37872; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73727269, jitterRate=0.0986219197511673}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:49,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:49,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872., pid=209, masterSystemTime=1676811529278 2023-02-19 12:58:49,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,310 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:49,310 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529310"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811529310"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811529310"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811529310"}]},"ts":"1676811529310"} 2023-02-19 12:58:49,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,312 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:49,312 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529312"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811529312"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811529312"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811529312"}]},"ts":"1676811529312"} 2023-02-19 12:58:49,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,313 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:49,313 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529313"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811529313"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811529313"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811529313"}]},"ts":"1676811529313"} 2023-02-19 12:58:49,314 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=204 2023-02-19 12:58:49,314 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=204, state=SUCCESS; OpenRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 187 msec 2023-02-19 12:58:49,315 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=206 2023-02-19 12:58:49,315 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=206, state=SUCCESS; OpenRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769 in 186 msec 2023-02-19 12:58:49,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN in 346 msec 2023-02-19 12:58:49,315 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=203 2023-02-19 12:58:49,316 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=203, state=SUCCESS; OpenRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696 in 188 msec 2023-02-19 12:58:49,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN in 347 msec 2023-02-19 12:58:49,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=203, resume processing ppid=202 2023-02-19 12:58:49,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN in 347 msec 2023-02-19 12:58:49,317 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-19 12:58:49,318 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811529317"}]},"ts":"1676811529317"} 2023-02-19 12:58:49,318 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=ENABLED in hbase:meta 2023-02-19 12:58:49,327 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 execute state=CREATE_TABLE_POST_OPERATION 2023-02-19 12:58:49,329 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676811528891 in 434 msec 2023-02-19 12:58:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-19 12:58:49,649 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 202 completed 2023-02-19 12:58:49,651 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:49,651 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,654 INFO [Listener at localhost/43399] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-19 12:58:49,666 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676811528891 2023-02-19 12:58:49,667 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676811528891 2023-02-19 12:58:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-19 12:58:49,670 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811529670"}]},"ts":"1676811529670"} 2023-02-19 12:58:49,671 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=DISABLING in hbase:meta 2023-02-19 12:58:49,672 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676811528891 to state=DISABLING 2023-02-19 12:58:49,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN}] 2023-02-19 12:58:49,674 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN 2023-02-19 12:58:49,675 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN 2023-02-19 12:58:49,675 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN 2023-02-19 12:58:49,675 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN 2023-02-19 12:58:49,675 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN 2023-02-19 12:58:49,675 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:49,675 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:49,675 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:49,675 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:49,675 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:49,676 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529675"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529675"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529675"}]},"ts":"1676811529675"} 2023-02-19 12:58:49,676 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529675"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529675"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529675"}]},"ts":"1676811529675"} 2023-02-19 12:58:49,676 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529675"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529675"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529675"}]},"ts":"1676811529675"} 2023-02-19 12:58:49,676 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529675"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529675"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529675"}]},"ts":"1676811529675"} 2023-02-19 12:58:49,676 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529675"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529675"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529675"}]},"ts":"1676811529675"} 2023-02-19 12:58:49,680 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=217, state=RUNNABLE; CloseRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:49,681 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=219, state=RUNNABLE; CloseRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:49,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=218, state=RUNNABLE; CloseRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:49,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=221, state=RUNNABLE; CloseRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:49,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=220, state=RUNNABLE; CloseRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:49,683 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN 2023-02-19 12:58:49,684 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:49,684 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529684"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811529684"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811529684"}]},"ts":"1676811529684"} 2023-02-19 12:58:49,686 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:49,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4aca2f2bf5da4c990f8539f674ac2e3e, disabling compactions & flushes 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 92751fca21674e6b7ab33630e2b6a119, disabling compactions & flushes 2023-02-19 12:58:49,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. after waiting 0 ms 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. after waiting 0 ms 2023-02-19 12:58:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9c525e8dcdbb974e61fda246b8d37872, disabling compactions & flushes 2023-02-19 12:58:49,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. after waiting 0 ms 2023-02-19 12:58:49,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:49,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:49,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:49,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:49,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:49,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:49,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:49,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:49,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:49,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:49,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c6d304ba41abeb8e6ae55cbd4c42f7c1, disabling compactions & flushes 2023-02-19 12:58:49,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. after waiting 0 ms 2023-02-19 12:58:49,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,839 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=CLOSED 2023-02-19 12:58:49,839 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811529838"}]},"ts":"1676811529838"} 2023-02-19 12:58:49,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:49,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9afa96a011d8f82b0bebbd3bac2f1fce, disabling compactions & flushes 2023-02-19 12:58:49,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. after waiting 0 ms 2023-02-19 12:58:49,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,839 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=CLOSED 2023-02-19 12:58:49,840 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529839"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811529839"}]},"ts":"1676811529839"} 2023-02-19 12:58:49,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:49,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0056208a08192d81a1d3f47ace8488e5, disabling compactions & flushes 2023-02-19 12:58:49,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. after waiting 0 ms 2023-02-19 12:58:49,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,841 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=CLOSED 2023-02-19 12:58:49,841 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529841"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811529841"}]},"ts":"1676811529841"} 2023-02-19 12:58:49,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:49,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:49,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:49,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:49,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:49,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:49,844 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=220 2023-02-19 12:58:49,844 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=220, state=SUCCESS; CloseRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737 in 158 msec 2023-02-19 12:58:49,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:49,844 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=218 2023-02-19 12:58:49,845 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=218, state=SUCCESS; CloseRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769 in 160 msec 2023-02-19 12:58:49,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-19 12:58:49,845 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=CLOSED 2023-02-19 12:58:49,845 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811529845"}]},"ts":"1676811529845"} 2023-02-19 12:58:49,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:49,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:49,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:49,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN in 171 msec 2023-02-19 12:58:49,846 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=CLOSED 2023-02-19 12:58:49,846 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811529846"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811529846"}]},"ts":"1676811529846"} 2023-02-19 12:58:49,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN in 172 msec 2023-02-19 12:58:49,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:49,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-19 12:58:49,848 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=CLOSED 2023-02-19 12:58:49,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696 in 158 msec 2023-02-19 12:58:49,848 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811529848"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811529848"}]},"ts":"1676811529848"} 2023-02-19 12:58:49,850 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=219 2023-02-19 12:58:49,850 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=219, state=SUCCESS; CloseRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769 in 166 msec 2023-02-19 12:58:49,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN in 175 msec 2023-02-19 12:58:49,850 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=217 2023-02-19 12:58:49,850 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=217, state=SUCCESS; CloseRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 167 msec 2023-02-19 12:58:49,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=221 2023-02-19 12:58:49,851 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN in 177 msec 2023-02-19 12:58:49,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN in 177 msec 2023-02-19 12:58:49,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=221, state=SUCCESS; CloseRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696 in 167 msec 2023-02-19 12:58:49,853 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=221, resume processing ppid=215 2023-02-19 12:58:49,853 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN in 178 msec 2023-02-19 12:58:49,854 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811529853"}]},"ts":"1676811529853"} 2023-02-19 12:58:49,854 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=DISABLED in hbase:meta 2023-02-19 12:58:49,856 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676811528891 to state=DISABLED 2023-02-19 12:58:49,858 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676811528891 in 190 msec 2023-02-19 12:58:49,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-19 12:58:49,921 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 215 completed 2023-02-19 12:58:49,922 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } 2023-02-19 12:58:49,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811529922 (current time:1676811529922). 2023-02-19 12:58:49,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:49,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x53ee22c5 to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:49,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76ddc8ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:49,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:49,931 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39596, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:49,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53ee22c5 to 127.0.0.1:60853 2023-02-19 12:58:49,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } 2023-02-19 12:58:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE 2023-02-19 12:58:49,934 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE 2023-02-19 12:58:49,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } 2023-02-19 12:58:49,935 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE; timeout=600000, timestamp=1676812129935 2023-02-19 12:58:49,935 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676811528891 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:49,935 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:49,935 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:49,948 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=DISABLED ttl=0 } 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,961 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,962 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,962 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,962 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,964 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,964 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,964 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:49,965 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:49,965 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:49,965 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:49,965 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:49,966 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:49,967 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:49,967 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:49,969 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-19 12:58:50,005 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:50,005 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676811528891 2023-02-19 12:58:50,005 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811528891 2023-02-19 12:58:50,023 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811528891 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/emptySnaptb-1676811528891 2023-02-19 12:58:50,041 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676811528891 of table testCorruptedSnapshot-1676811528891 completed 2023-02-19 12:58:50,041 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811528891 2023-02-19 12:58:50,041 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/emptySnaptb-1676811528891 2023-02-19 12:58:50,041 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676811528891 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676811528891 at 1676811529935Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=DISABLED ttl=0 } at 1676811529949 (+14 ms)Consolidate snapshot: emptySnaptb-1676811528891 at 1676811530005 (+56 ms)Loading Region manifests for emptySnaptb-1676811528891 at 1676811530005Writing data manifest for emptySnaptb-1676811528891 at 1676811530009 (+4 ms)Verifying snapshot: emptySnaptb-1676811528891 at 1676811530018 (+9 ms)Snapshot emptySnaptb-1676811528891 of table testCorruptedSnapshot-1676811528891 completed at 1676811530041 (+23 ms) 2023-02-19 12:58:50,044 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE 2023-02-19 12:58:50,045 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE in 111 msec 2023-02-19 12:58:50,185 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:50,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } is done 2023-02-19 12:58:50,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:50,187 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,187 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-19 12:58:50,190 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,191 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811530191"}]},"ts":"1676811530191"} 2023-02-19 12:58:50,192 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=ENABLING in hbase:meta 2023-02-19 12:58:50,194 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:50,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN}] 2023-02-19 12:58:50,195 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN 2023-02-19 12:58:50,195 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN 2023-02-19 12:58:50,195 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN 2023-02-19 12:58:50,195 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN 2023-02-19 12:58:50,196 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,196 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,196 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,196 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,196 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:50,196 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN 2023-02-19 12:58:50,196 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:50,196 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:50,196 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:50,197 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,197 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN 2023-02-19 12:58:50,197 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:50,198 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,198 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:50,346 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:50,347 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,347 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,347 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,347 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,347 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811530347"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530347"}]},"ts":"1676811530347"} 2023-02-19 12:58:50,347 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530347"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530347"}]},"ts":"1676811530347"} 2023-02-19 12:58:50,347 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,347 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811530347"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530347"}]},"ts":"1676811530347"} 2023-02-19 12:58:50,347 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530347"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530347"}]},"ts":"1676811530347"} 2023-02-19 12:58:50,347 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530347"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530347"}]},"ts":"1676811530347"} 2023-02-19 12:58:50,349 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=233, state=RUNNABLE; OpenRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:50,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=235, state=RUNNABLE; OpenRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:50,350 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=230, state=RUNNABLE; OpenRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:50,350 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=232, state=RUNNABLE; OpenRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:50,351 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=231, state=RUNNABLE; OpenRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:50,352 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,352 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530352"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530352"}]},"ts":"1676811530352"} 2023-02-19 12:58:50,353 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=234, state=RUNNABLE; OpenRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-19 12:58:50,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:50,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 92751fca21674e6b7ab33630e2b6a119, NAME => 'testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:50,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:50,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:50,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0056208a08192d81a1d3f47ace8488e5, NAME => 'testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:50,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:50,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,505 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:50,506 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf 2023-02-19 12:58:50,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4aca2f2bf5da4c990f8539f674ac2e3e, NAME => 'testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:50,506 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf 2023-02-19 12:58:50,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:50,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,507 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92751fca21674e6b7ab33630e2b6a119 columnFamilyName cf 2023-02-19 12:58:50,507 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,507 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(310): Store=92751fca21674e6b7ab33630e2b6a119/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:50,508 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf 2023-02-19 12:58:50,508 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf 2023-02-19 12:58:50,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,508 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4aca2f2bf5da4c990f8539f674ac2e3e columnFamilyName cf 2023-02-19 12:58:50,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,509 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(310): Store=4aca2f2bf5da4c990f8539f674ac2e3e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:50,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:50,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 92751fca21674e6b7ab33630e2b6a119; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70461919, jitterRate=0.04996441304683685}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:50,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:50,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119., pid=239, masterSystemTime=1676811530501 2023-02-19 12:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:50,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:50,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9afa96a011d8f82b0bebbd3bac2f1fce, NAME => 'testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,515 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,516 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530515"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811530515"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811530515"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811530515"}]},"ts":"1676811530515"} 2023-02-19 12:58:50,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:50,517 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,517 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4aca2f2bf5da4c990f8539f674ac2e3e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65238311, jitterRate=-0.027873411774635315}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:50,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:50,518 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf 2023-02-19 12:58:50,518 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf 2023-02-19 12:58:50,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e., pid=241, masterSystemTime=1676811530503 2023-02-19 12:58:50,519 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9afa96a011d8f82b0bebbd3bac2f1fce columnFamilyName cf 2023-02-19 12:58:50,519 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=232 2023-02-19 12:58:50,520 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=232, state=SUCCESS; OpenRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769 in 167 msec 2023-02-19 12:58:50,520 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(310): Store=9afa96a011d8f82b0bebbd3bac2f1fce/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:50,520 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:50,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:50,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:50,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c6d304ba41abeb8e6ae55cbd4c42f7c1, NAME => 'testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:50,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:50,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,521 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,521 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530521"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811530521"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811530521"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811530521"}]},"ts":"1676811530521"} 2023-02-19 12:58:50,522 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN in 326 msec 2023-02-19 12:58:50,522 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,523 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf 2023-02-19 12:58:50,523 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf 2023-02-19 12:58:50,524 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6d304ba41abeb8e6ae55cbd4c42f7c1 columnFamilyName cf 2023-02-19 12:58:50,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=234 2023-02-19 12:58:50,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=234, state=SUCCESS; OpenRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737 in 170 msec 2023-02-19 12:58:50,525 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(310): Store=c6d304ba41abeb8e6ae55cbd4c42f7c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:50,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:50,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN in 331 msec 2023-02-19 12:58:50,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9afa96a011d8f82b0bebbd3bac2f1fce; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69294233, jitterRate=0.03256453573703766}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:50,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:50,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce., pid=236, masterSystemTime=1676811530501 2023-02-19 12:58:50,528 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf 2023-02-19 12:58:50,528 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf 2023-02-19 12:58:50,529 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0056208a08192d81a1d3f47ace8488e5 columnFamilyName cf 2023-02-19 12:58:50,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:50,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:50,529 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,529 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530529"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811530529"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811530529"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811530529"}]},"ts":"1676811530529"} 2023-02-19 12:58:50,530 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(310): Store=0056208a08192d81a1d3f47ace8488e5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:50,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:50,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c6d304ba41abeb8e6ae55cbd4c42f7c1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61076391, jitterRate=-0.08989085257053375}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:50,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:50,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1., pid=240, masterSystemTime=1676811530503 2023-02-19 12:58:50,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=233 2023-02-19 12:58:50,533 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=233, state=SUCCESS; OpenRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769 in 182 msec 2023-02-19 12:58:50,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:50,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:50,534 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN in 339 msec 2023-02-19 12:58:50,534 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530534"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811530534"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811530534"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811530534"}]},"ts":"1676811530534"} 2023-02-19 12:58:50,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:50,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0056208a08192d81a1d3f47ace8488e5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72845350, jitterRate=0.08548030257225037}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:50,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:50,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5., pid=237, masterSystemTime=1676811530502 2023-02-19 12:58:50,536 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=231 2023-02-19 12:58:50,537 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=231, state=SUCCESS; OpenRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 184 msec 2023-02-19 12:58:50,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:50,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:50,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:50,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9c525e8dcdbb974e61fda246b8d37872, NAME => 'testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:50,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:50,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN in 343 msec 2023-02-19 12:58:50,538 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,538 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811530538"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811530538"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811530538"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811530538"}]},"ts":"1676811530538"} 2023-02-19 12:58:50,539 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,540 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf 2023-02-19 12:58:50,540 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf 2023-02-19 12:58:50,540 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9c525e8dcdbb974e61fda246b8d37872 columnFamilyName cf 2023-02-19 12:58:50,541 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=235 2023-02-19 12:58:50,541 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=235, state=SUCCESS; OpenRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696 in 191 msec 2023-02-19 12:58:50,541 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(310): Store=9c525e8dcdbb974e61fda246b8d37872/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:50,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,543 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN in 347 msec 2023-02-19 12:58:50,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:50,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9c525e8dcdbb974e61fda246b8d37872; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70032682, jitterRate=0.04356828331947327}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:50,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:50,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872., pid=238, masterSystemTime=1676811530502 2023-02-19 12:58:50,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:50,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:50,547 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,548 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811530547"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811530547"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811530547"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811530547"}]},"ts":"1676811530547"} 2023-02-19 12:58:50,550 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=230 2023-02-19 12:58:50,550 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=230, state=SUCCESS; OpenRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696 in 199 msec 2023-02-19 12:58:50,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=230, resume processing ppid=229 2023-02-19 12:58:50,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN in 356 msec 2023-02-19 12:58:50,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811530552"}]},"ts":"1676811530552"} 2023-02-19 12:58:50,553 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=ENABLED in hbase:meta 2023-02-19 12:58:50,555 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676811528891' was successfully enabled. 2023-02-19 12:58:50,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676811528891 in 368 msec 2023-02-19 12:58:50,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-19 12:58:50,942 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 229 completed 2023-02-19 12:58:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:50,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:50,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:50,957 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,957 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:50,957 INFO [Listener at localhost/43399] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-19 12:58:50,968 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:50,969 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:50,970 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:50,971 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:50,972 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:50,975 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,975 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-19 12:58:50,979 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811530979"}]},"ts":"1676811530979"} 2023-02-19 12:58:50,981 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=DISABLING in hbase:meta 2023-02-19 12:58:50,984 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676811528891 to state=DISABLING 2023-02-19 12:58:50,985 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN}] 2023-02-19 12:58:50,986 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN 2023-02-19 12:58:50,986 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN 2023-02-19 12:58:50,986 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN 2023-02-19 12:58:50,986 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN 2023-02-19 12:58:50,986 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN 2023-02-19 12:58:50,986 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,986 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,986 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:50,987 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811530986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530986"}]},"ts":"1676811530986"} 2023-02-19 12:58:50,987 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530986"}]},"ts":"1676811530986"} 2023-02-19 12:58:50,987 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530986"}]},"ts":"1676811530986"} 2023-02-19 12:58:50,987 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,987 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:50,987 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530987"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530987"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530987"}]},"ts":"1676811530987"} 2023-02-19 12:58:50,987 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811530987"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530987"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530987"}]},"ts":"1676811530987"} 2023-02-19 12:58:50,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=243, state=RUNNABLE; CloseRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:50,989 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=245, state=RUNNABLE; CloseRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:50,990 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=246, state=RUNNABLE; CloseRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:50,991 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=244, state=RUNNABLE; CloseRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:50,991 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=248, state=RUNNABLE; CloseRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:50,992 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN 2023-02-19 12:58:50,992 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:50,993 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811530992"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811530992"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811530992"}]},"ts":"1676811530992"} 2023-02-19 12:58:50,994 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:51,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:51,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 92751fca21674e6b7ab33630e2b6a119, disabling compactions & flushes 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9c525e8dcdbb974e61fda246b8d37872, disabling compactions & flushes 2023-02-19 12:58:51,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:51,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. after waiting 0 ms 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. after waiting 0 ms 2023-02-19 12:58:51,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:51,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9c525e8dcdbb974e61fda246b8d37872 1/1 column families, dataSize=2.75 KB heapSize=6.16 KB 2023-02-19 12:58:51,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 92751fca21674e6b7ab33630e2b6a119 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-19 12:58:51,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:51,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4aca2f2bf5da4c990f8539f674ac2e3e, disabling compactions & flushes 2023-02-19 12:58:51,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:51,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:51,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. after waiting 0 ms 2023-02-19 12:58:51,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:51,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 4aca2f2bf5da4c990f8539f674ac2e3e 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-19 12:58:51,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230219266d8e9613bd43069140deb8608a4f00 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219266d8e9613bd43069140deb8608a4f00 2023-02-19 12:58:51,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021935bdd2b1ff89445fb3532b37fe28a778 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021935bdd2b1ff89445fb3532b37fe28a778 2023-02-19 12:58:51,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230219712c20746b9441edaab22fb6c533ddb6 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e20230219712c20746b9441edaab22fb6c533ddb6 2023-02-19 12:58:51,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/.tmp/cf/6906358dd2884f3ca1e06713719c580d 2023-02-19 12:58:51,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/.tmp/cf/18895ddcab074cc6a6095930dcf5d793 2023-02-19 12:58:51,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/.tmp/cf/01fdd656a5fa4385bf5633fe0a572a69 2023-02-19 12:58:51,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/.tmp/cf/6906358dd2884f3ca1e06713719c580d as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d 2023-02-19 12:58:51,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/.tmp/cf/18895ddcab074cc6a6095930dcf5d793 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793 2023-02-19 12:58:51,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/.tmp/cf/01fdd656a5fa4385bf5633fe0a572a69 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69 2023-02-19 12:58:51,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d, entries=28, sequenceid=8, filesize=9.7 K 2023-02-19 12:58:51,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 4aca2f2bf5da4c990f8539f674ac2e3e in 41ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:51,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676811528891' 2023-02-19 12:58:51,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793, entries=42, sequenceid=8, filesize=12.1 K 2023-02-19 12:58:51,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69, entries=29, sequenceid=8, filesize=9.9 K 2023-02-19 12:58:51,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.75 KB/2812, heapSize ~6.14 KB/6288, currentSize=0 B/0 for 9c525e8dcdbb974e61fda246b8d37872 in 46ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:51,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 92751fca21674e6b7ab33630e2b6a119 in 48ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:51,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:51,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:51,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:51,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c6d304ba41abeb8e6ae55cbd4c42f7c1, disabling compactions & flushes 2023-02-19 12:58:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. after waiting 0 ms 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:51,194 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=CLOSED 2023-02-19 12:58:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c6d304ba41abeb8e6ae55cbd4c42f7c1 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-19 12:58:51,194 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811531194"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811531194"}]},"ts":"1676811531194"} 2023-02-19 12:58:51,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:51,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:51,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:51,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0056208a08192d81a1d3f47ace8488e5, disabling compactions & flushes 2023-02-19 12:58:51,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:51,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:51,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. after waiting 0 ms 2023-02-19 12:58:51,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:51,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0056208a08192d81a1d3f47ace8488e5 1/1 column families, dataSize=22.57 KB heapSize=48.77 KB 2023-02-19 12:58:51,196 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=CLOSED 2023-02-19 12:58:51,197 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811531196"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811531196"}]},"ts":"1676811531196"} 2023-02-19 12:58:51,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:51,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:51,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9afa96a011d8f82b0bebbd3bac2f1fce, disabling compactions & flushes 2023-02-19 12:58:51,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:51,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:51,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. after waiting 0 ms 2023-02-19 12:58:51,197 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=CLOSED 2023-02-19 12:58:51,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:51,197 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811531197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811531197"}]},"ts":"1676811531197"} 2023-02-19 12:58:51,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9afa96a011d8f82b0bebbd3bac2f1fce 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-19 12:58:51,207 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-19 12:58:51,207 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737 in 203 msec 2023-02-19 12:58:51,207 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=243 2023-02-19 12:58:51,208 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=243, state=SUCCESS; CloseRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696 in 210 msec 2023-02-19 12:58:51,208 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=245 2023-02-19 12:58:51,208 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN in 222 msec 2023-02-19 12:58:51,208 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=245, state=SUCCESS; CloseRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769 in 213 msec 2023-02-19 12:58:51,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN in 222 msec 2023-02-19 12:58:51,209 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN in 223 msec 2023-02-19 12:58:51,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230219e08f0f0bb3aa44b7ad373721f2beae45 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219e08f0f0bb3aa44b7ad373721f2beae45 2023-02-19 12:58:51,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230219c842bb5938004f4590c55a77b9cf255d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d520230219c842bb5938004f4590c55a77b9cf255d 2023-02-19 12:58:51,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/.tmp/cf/f7abb3c68c6c4328affbf68d38caa446 2023-02-19 12:58:51,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/.tmp/cf/f7abb3c68c6c4328affbf68d38caa446 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446 2023-02-19 12:58:51,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-19 12:58:51,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446, entries=23, sequenceid=8, filesize=8.8 K 2023-02-19 12:58:51,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 9afa96a011d8f82b0bebbd3bac2f1fce in 35ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:51,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:51,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/.tmp/cf/2e493bcf8bc04760a2b49e2aa01d7135 2023-02-19 12:58:51,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:51,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:51,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:51,241 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=CLOSED 2023-02-19 12:58:51,241 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811531241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811531241"}]},"ts":"1676811531241"} 2023-02-19 12:58:51,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/.tmp/cf/2e493bcf8bc04760a2b49e2aa01d7135 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135 2023-02-19 12:58:51,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=246 2023-02-19 12:58:51,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=246, state=SUCCESS; CloseRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769 in 252 msec 2023-02-19 12:58:51,245 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN in 259 msec 2023-02-19 12:58:51,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135, entries=345, sequenceid=8, filesize=64.4 K 2023-02-19 12:58:51,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for 0056208a08192d81a1d3f47ace8488e5 in 50ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:51,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:51,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:51,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:51,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:51,250 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=CLOSED 2023-02-19 12:58:51,250 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811531250"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811531250"}]},"ts":"1676811531250"} 2023-02-19 12:58:51,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=248 2023-02-19 12:58:51,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=248, state=SUCCESS; CloseRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696 in 260 msec 2023-02-19 12:58:51,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN in 267 msec 2023-02-19 12:58:51,615 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230219731fc428f5da482391cbbca8974cad47 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219731fc428f5da482391cbbca8974cad47 2023-02-19 12:58:51,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/.tmp/cf/cba708de7bc749579ba8d4268b7bb26d 2023-02-19 12:58:51,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/.tmp/cf/cba708de7bc749579ba8d4268b7bb26d as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d 2023-02-19 12:58:51,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d, entries=33, sequenceid=8, filesize=10.6 K 2023-02-19 12:58:51,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for c6d304ba41abeb8e6ae55cbd4c42f7c1 in 436ms, sequenceid=8, compaction requested=false 2023-02-19 12:58:51,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-19 12:58:51,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:51,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:51,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:51,635 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=CLOSED 2023-02-19 12:58:51,635 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811531635"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811531635"}]},"ts":"1676811531635"} 2023-02-19 12:58:51,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=244 2023-02-19 12:58:51,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=244, state=SUCCESS; CloseRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 645 msec 2023-02-19 12:58:51,638 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=244, resume processing ppid=242 2023-02-19 12:58:51,638 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN in 652 msec 2023-02-19 12:58:51,639 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811531639"}]},"ts":"1676811531639"} 2023-02-19 12:58:51,640 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=DISABLED in hbase:meta 2023-02-19 12:58:51,642 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676811528891 to state=DISABLED 2023-02-19 12:58:51,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676811528891 in 667 msec 2023-02-19 12:58:51,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-19 12:58:51,731 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 242 completed 2023-02-19 12:58:51,732 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } 2023-02-19 12:58:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676811531732 (current time:1676811531732). 2023-02-19 12:58:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-19 12:58:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(139): Connect 0x733b3c3d to 127.0.0.1:60853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-19 12:58:51,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51e1d095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-19 12:58:51,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-19 12:58:51,741 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39600, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-19 12:58:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x733b3c3d to 127.0.0.1:60853 2023-02-19 12:58:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-19 12:58:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } 2023-02-19 12:58:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE 2023-02-19 12:58:51,744 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE 2023-02-19 12:58:51,745 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE; timeout=600000, timestamp=1676812131745 2023-02-19 12:58:51,745 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676811528891 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:51,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } 2023-02-19 12:58:51,746 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-19 12:58:51,746 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-19 12:58:51,753 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=DISABLED ttl=0 } 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-19 12:58:51,755 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,756 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69] hfiles 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d] hfiles 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446] hfiles 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135] hfiles 2023-02-19 12:58:51,757 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135 2023-02-19 12:58:51,758 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793] hfiles 2023-02-19 12:58:51,758 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021935bdd2b1ff89445fb3532b37fe28a778, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219731fc428f5da482391cbbca8974cad47, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219266d8e9613bd43069140deb8608a4f00, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e20230219712c20746b9441edaab22fb6c533ddb6, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d520230219c842bb5938004f4590c55a77b9cf255d, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219e08f0f0bb3aa44b7ad373721f2beae45] hfiles 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021935bdd2b1ff89445fb3532b37fe28a778 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219731fc428f5da482391cbbca8974cad47 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219266d8e9613bd43069140deb8608a4f00 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e20230219712c20746b9441edaab22fb6c533ddb6 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d520230219c842bb5938004f4590c55a77b9cf255d 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219e08f0f0bb3aa44b7ad373721f2beae45 2023-02-19 12:58:51,759 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d] hfiles 2023-02-19 12:58:51,760 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d 2023-02-19 12:58:51,996 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:51,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } is done 2023-02-19 12:58:51,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 }' is still in progress! 2023-02-19 12:58:51,997 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-19 12:58:52,184 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=DISABLED ttl=0 } as finished. 2023-02-19 12:58:52,184 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676811528891 2023-02-19 12:58:52,185 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811528891 2023-02-19 12:58:52,202 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811528891 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/snaptb0-1676811528891 2023-02-19 12:58:52,220 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676811528891 of table testCorruptedSnapshot-1676811528891 completed 2023-02-19 12:58:52,220 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811528891 2023-02-19 12:58:52,220 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.hbase-snapshot/.tmp/snaptb0-1676811528891 2023-02-19 12:58:52,220 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676811528891 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676811528891 at 1676811531745Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=DISABLED ttl=0 } at 1676811531754 (+9 ms)Consolidate snapshot: snaptb0-1676811528891 at 1676811532184 (+430 ms)Loading Region manifests for snaptb0-1676811528891 at 1676811532184Writing data manifest for snaptb0-1676811528891 at 1676811532188 (+4 ms)Verifying snapshot: snaptb0-1676811528891 at 1676811532195 (+7 ms)Snapshot snaptb0-1676811528891 of table testCorruptedSnapshot-1676811528891 completed at 1676811532220 (+25 ms) 2023-02-19 12:58:52,221 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE 2023-02-19 12:58:52,222 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676811528891, type=EXCLUSIVE in 478 msec 2023-02-19 12:58:52,497 DEBUG [Listener at localhost/43399] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-19 12:58:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 } is done 2023-02-19 12:58:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676811528891 table=testCorruptedSnapshot-1676811528891 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-19 12:58:52,498 INFO [Listener at localhost/43399] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676811528891 2023-02-19 12:58:52,499 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676811528891 2023-02-19 12:58:52,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:52,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-19 12:58:52,502 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:52,503 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811532502"}]},"ts":"1676811532502"} 2023-02-19 12:58:52,504 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=ENABLING in hbase:meta 2023-02-19 12:58:52,507 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-19 12:58:52,507 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN}] 2023-02-19 12:58:52,507 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN 2023-02-19 12:58:52,508 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN 2023-02-19 12:58:52,508 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN 2023-02-19 12:58:52,508 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN 2023-02-19 12:58:52,508 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:52,508 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:52,508 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN 2023-02-19 12:58:52,508 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:52,508 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:52,508 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,45077,1676811501769; forceNewPlan=false, retain=true 2023-02-19 12:58:52,508 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:52,509 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:52,509 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:52,509 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:52,510 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41307,1676811501696; forceNewPlan=false, retain=true 2023-02-19 12:58:52,510 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN 2023-02-19 12:58:52,514 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:52,514 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39199,1676811501737; forceNewPlan=false, retain=true 2023-02-19 12:58:52,659 INFO [jenkins-hbase4:34259] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-19 12:58:52,659 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:52,659 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:52,659 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:52,659 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811532659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811532659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811532659"}]},"ts":"1676811532659"} 2023-02-19 12:58:52,659 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:52,659 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:52,659 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811532659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811532659"}]},"ts":"1676811532659"} 2023-02-19 12:58:52,659 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811532659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811532659"}]},"ts":"1676811532659"} 2023-02-19 12:58:52,659 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811532659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811532659"}]},"ts":"1676811532659"} 2023-02-19 12:58:52,659 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811532659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811532659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811532659"}]},"ts":"1676811532659"} 2023-02-19 12:58:52,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=262, state=RUNNABLE; OpenRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:52,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=258, state=RUNNABLE; OpenRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:52,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=260, state=RUNNABLE; OpenRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:52,663 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:52,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=257, state=RUNNABLE; OpenRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:52,664 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:52,664 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532664"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811532664"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811532664"}]},"ts":"1676811532664"} 2023-02-19 12:58:52,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=261, state=RUNNABLE; OpenRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:52,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-19 12:58:52,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:52,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:52,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9c525e8dcdbb974e61fda246b8d37872, NAME => 'testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.', STARTKEY => '', ENDKEY => '1'} 2023-02-19 12:58:52,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c6d304ba41abeb8e6ae55cbd4c42f7c1, NAME => 'testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.', STARTKEY => '1', ENDKEY => '2'} 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9afa96a011d8f82b0bebbd3bac2f1fce, NAME => 'testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.', STARTKEY => '3', ENDKEY => '4'} 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:52,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,817 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,817 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,818 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,818 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf 2023-02-19 12:58:52,818 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf 2023-02-19 12:58:52,818 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf 2023-02-19 12:58:52,819 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf 2023-02-19 12:58:52,819 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6d304ba41abeb8e6ae55cbd4c42f7c1 columnFamilyName cf 2023-02-19 12:58:52,819 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9c525e8dcdbb974e61fda246b8d37872 columnFamilyName cf 2023-02-19 12:58:52,819 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf 2023-02-19 12:58:52,819 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf 2023-02-19 12:58:52,819 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9afa96a011d8f82b0bebbd3bac2f1fce columnFamilyName cf 2023-02-19 12:58:52,826 DEBUG [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793 2023-02-19 12:58:52,826 INFO [StoreOpener-9c525e8dcdbb974e61fda246b8d37872-1] regionserver.HStore(310): Store=9c525e8dcdbb974e61fda246b8d37872/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:52,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,827 DEBUG [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d 2023-02-19 12:58:52,827 DEBUG [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446 2023-02-19 12:58:52,827 INFO [StoreOpener-c6d304ba41abeb8e6ae55cbd4c42f7c1-1] regionserver.HStore(310): Store=c6d304ba41abeb8e6ae55cbd4c42f7c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:52,827 INFO [StoreOpener-9afa96a011d8f82b0bebbd3bac2f1fce-1] regionserver.HStore(310): Store=9afa96a011d8f82b0bebbd3bac2f1fce/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:52,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:52,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9c525e8dcdbb974e61fda246b8d37872; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68633428, jitterRate=0.022717773914337158}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:52,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:52,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:52,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:52,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872., pid=267, masterSystemTime=1676811532813 2023-02-19 12:58:52,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c6d304ba41abeb8e6ae55cbd4c42f7c1; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67706972, jitterRate=0.008912503719329834}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:52,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:52,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9afa96a011d8f82b0bebbd3bac2f1fce; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67142405, jitterRate=4.997998476028442E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:52,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:52,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1., pid=264, masterSystemTime=1676811532813 2023-02-19 12:58:52,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:52,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:52,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0056208a08192d81a1d3f47ace8488e5, NAME => 'testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.', STARTKEY => '5', ENDKEY => ''} 2023-02-19 12:58:52,836 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,836 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811532836"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811532836"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811532836"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811532836"}]},"ts":"1676811532836"} 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:52,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4aca2f2bf5da4c990f8539f674ac2e3e, NAME => 'testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.', STARTKEY => '4', ENDKEY => '5'} 2023-02-19 12:58:52,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce., pid=265, masterSystemTime=1676811532814 2023-02-19 12:58:52,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:52,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,837 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:52,837 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532837"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811532837"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811532837"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811532837"}]},"ts":"1676811532837"} 2023-02-19 12:58:52,838 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,838 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:52,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:52,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:52,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 92751fca21674e6b7ab33630e2b6a119, NAME => 'testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.', STARTKEY => '2', ENDKEY => '3'} 2023-02-19 12:58:52,839 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf 2023-02-19 12:58:52,839 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf 2023-02-19 12:58:52,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676811528891 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-19 12:58:52,839 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf 2023-02-19 12:58:52,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,840 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:52,840 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532840"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811532840"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811532840"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811532840"}]},"ts":"1676811532840"} 2023-02-19 12:58:52,840 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf 2023-02-19 12:58:52,841 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0056208a08192d81a1d3f47ace8488e5 columnFamilyName cf 2023-02-19 12:58:52,841 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4aca2f2bf5da4c990f8539f674ac2e3e columnFamilyName cf 2023-02-19 12:58:52,845 DEBUG [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135 2023-02-19 12:58:52,845 INFO [StoreOpener-0056208a08192d81a1d3f47ace8488e5-1] regionserver.HStore(310): Store=0056208a08192d81a1d3f47ace8488e5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:52,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=257 2023-02-19 12:58:52,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=257, state=SUCCESS; OpenRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696 in 175 msec 2023-02-19 12:58:52,847 DEBUG [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d 2023-02-19 12:58:52,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,848 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=258 2023-02-19 12:58:52,848 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=258, state=SUCCESS; OpenRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 178 msec 2023-02-19 12:58:52,848 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=260 2023-02-19 12:58:52,848 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=260, state=SUCCESS; OpenRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769 in 179 msec 2023-02-19 12:58:52,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, ASSIGN in 340 msec 2023-02-19 12:58:52,849 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, ASSIGN in 341 msec 2023-02-19 12:58:52,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, ASSIGN in 341 msec 2023-02-19 12:58:52,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:52,852 INFO [StoreOpener-4aca2f2bf5da4c990f8539f674ac2e3e-1] regionserver.HStore(310): Store=4aca2f2bf5da4c990f8539f674ac2e3e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:52,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0056208a08192d81a1d3f47ace8488e5; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62381697, jitterRate=-0.07044027745723724}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:52,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:52,853 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5., pid=263, masterSystemTime=1676811532813 2023-02-19 12:58:52,854 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf 2023-02-19 12:58:52,854 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf 2023-02-19 12:58:52,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,855 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92751fca21674e6b7ab33630e2b6a119 columnFamilyName cf 2023-02-19 12:58:52,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:52,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:52,855 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:52,856 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811532855"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811532855"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811532855"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811532855"}]},"ts":"1676811532855"} 2023-02-19 12:58:52,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:52,859 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=262 2023-02-19 12:58:52,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4aca2f2bf5da4c990f8539f674ac2e3e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70288224, jitterRate=0.047376155853271484}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:52,859 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=262, state=SUCCESS; OpenRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696 in 196 msec 2023-02-19 12:58:52,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:52,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e., pid=268, masterSystemTime=1676811532813 2023-02-19 12:58:52,860 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, ASSIGN in 352 msec 2023-02-19 12:58:52,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:52,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:52,861 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:52,861 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532861"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811532861"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811532861"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811532861"}]},"ts":"1676811532861"} 2023-02-19 12:58:52,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=261 2023-02-19 12:58:52,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=261, state=SUCCESS; OpenRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737 in 196 msec 2023-02-19 12:58:52,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, ASSIGN in 357 msec 2023-02-19 12:58:52,867 DEBUG [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(539): loaded hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69 2023-02-19 12:58:52,867 INFO [StoreOpener-92751fca21674e6b7ab33630e2b6a119-1] regionserver.HStore(310): Store=92751fca21674e6b7ab33630e2b6a119/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-19 12:58:52,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:52,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 92751fca21674e6b7ab33630e2b6a119; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71508137, jitterRate=0.06555427610874176}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-19 12:58:52,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:52,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119., pid=266, masterSystemTime=1676811532814 2023-02-19 12:58:52,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:52,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:52,874 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:52,875 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811532874"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676811532874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676811532874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676811532874"}]},"ts":"1676811532874"} 2023-02-19 12:58:52,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-19 12:58:52,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769 in 213 msec 2023-02-19 12:58:52,878 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=259, resume processing ppid=256 2023-02-19 12:58:52,878 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, ASSIGN in 370 msec 2023-02-19 12:58:52,879 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811532879"}]},"ts":"1676811532879"} 2023-02-19 12:58:52,880 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=ENABLED in hbase:meta 2023-02-19 12:58:52,882 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676811528891' was successfully enabled. 2023-02-19 12:58:52,884 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676811528891 in 384 msec 2023-02-19 12:58:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-19 12:58:53,253 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 256 completed 2023-02-19 12:58:53,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:53,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:53,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:53,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39199] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:53,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:53,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41307] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-19 12:58:53,264 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676811528891 2023-02-19 12:58:53,265 DEBUG [Listener at localhost/43399] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:53,265 INFO [Listener at localhost/43399] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-19 12:58:53,393 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-19 12:58:53,466 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-19 12:58:53,532 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-19 12:58:53,584 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-19 12:58:53,648 DEBUG [Listener at localhost/43399] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-19 12:58:53,897 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e1138f2 to 127.0.0.1:60853 2023-02-19 12:58:53,897 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:54,007 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-19 12:58:54,472 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 2e493bcf8bc04760a2b49e2aa01d7135 in the real (hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135) or archive (hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135) directory for the primary table. 2023-02-19 12:58:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 2e493bcf8bc04760a2b49e2aa01d7135 in the real (hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135) or archive (hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2506) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2502) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-19 12:58:54,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] ipc.CallRunner(144): callId: 362 service: MasterService methodName: RestoreSnapshot size: 116 connection: 172.31.14.131:40180 deadline: 1676811594468, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 2e493bcf8bc04760a2b49e2aa01d7135 in the real (hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135) or archive (hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/2e493bcf8bc04760a2b49e2aa01d7135) directory for the primary table. 2023-02-19 12:58:54,492 INFO [Listener at localhost/43399] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676811528891 2023-02-19 12:58:54,493 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676811528891 2023-02-19 12:58:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-19 12:58:54,496 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811534496"}]},"ts":"1676811534496"} 2023-02-19 12:58:54,497 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=DISABLING in hbase:meta 2023-02-19 12:58:54,498 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676811528891 to state=DISABLING 2023-02-19 12:58:54,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN}] 2023-02-19 12:58:54,499 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN 2023-02-19 12:58:54,499 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN 2023-02-19 12:58:54,499 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN 2023-02-19 12:58:54,499 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN 2023-02-19 12:58:54,500 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN 2023-02-19 12:58:54,500 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:54,500 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811534500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811534500"}]},"ts":"1676811534500"} 2023-02-19 12:58:54,500 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:54,500 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:54,500 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811534500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811534500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811534500"}]},"ts":"1676811534500"} 2023-02-19 12:58:54,500 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811534500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811534500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811534500"}]},"ts":"1676811534500"} 2023-02-19 12:58:54,500 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:54,500 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:54,501 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811534500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811534500"}]},"ts":"1676811534500"} 2023-02-19 12:58:54,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811534500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811534500"}]},"ts":"1676811534500"} 2023-02-19 12:58:54,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=274, state=RUNNABLE; CloseRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:54,501 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=271, state=RUNNABLE; CloseRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:54,502 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=276, state=RUNNABLE; CloseRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696}] 2023-02-19 12:58:54,502 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=272, state=RUNNABLE; CloseRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:54,503 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=273, state=RUNNABLE; CloseRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769}] 2023-02-19 12:58:54,504 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN 2023-02-19 12:58:54,504 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:54,505 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534504"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676811534504"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676811534504"}]},"ts":"1676811534504"} 2023-02-19 12:58:54,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=275, state=RUNNABLE; CloseRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737}] 2023-02-19 12:58:54,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:54,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 92751fca21674e6b7ab33630e2b6a119, disabling compactions & flushes 2023-02-19 12:58:54,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:54,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:54,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. after waiting 0 ms 2023-02-19 12:58:54,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:54,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 92751fca21674e6b7ab33630e2b6a119 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-19 12:58:54,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9c525e8dcdbb974e61fda246b8d37872, disabling compactions & flushes 2023-02-19 12:58:54,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. after waiting 0 ms 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:54,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9c525e8dcdbb974e61fda246b8d37872 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-19 12:58:54,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c6d304ba41abeb8e6ae55cbd4c42f7c1, disabling compactions & flushes 2023-02-19 12:58:54,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. after waiting 0 ms 2023-02-19 12:58:54,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:54,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c6d304ba41abeb8e6ae55cbd4c42f7c1 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-19 12:58:54,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230219805b5aff2d5e4304b829a115b84beb78 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219805b5aff2d5e4304b829a115b84beb78 2023-02-19 12:58:54,674 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302196d3aa93b6dc54d0db23f5e67224fbec9 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e202302196d3aa93b6dc54d0db23f5e67224fbec9 2023-02-19 12:58:54,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/.tmp/cf/0be92ab043194f2383f6e3a9c100e97e 2023-02-19 12:58:54,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/.tmp/cf/374a57461bb74402a4acf0c0242ddc36 2023-02-19 12:58:54,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/.tmp/cf/0be92ab043194f2383f6e3a9c100e97e as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/0be92ab043194f2383f6e3a9c100e97e 2023-02-19 12:58:54,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/.tmp/cf/374a57461bb74402a4acf0c0242ddc36 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/374a57461bb74402a4acf0c0242ddc36 2023-02-19 12:58:54,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230219c98e9edb9ed1442a957e4e318b8431ab to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219c98e9edb9ed1442a957e4e318b8431ab 2023-02-19 12:58:54,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/374a57461bb74402a4acf0c0242ddc36, entries=35, sequenceid=15, filesize=10.9 K 2023-02-19 12:58:54,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/0be92ab043194f2383f6e3a9c100e97e, entries=31, sequenceid=15, filesize=10.2 K 2023-02-19 12:58:54,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 92751fca21674e6b7ab33630e2b6a119 in 35ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:54,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 9c525e8dcdbb974e61fda246b8d37872 in 35ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:54,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:54,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:54,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872. 2023-02-19 12:58:54,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9c525e8dcdbb974e61fda246b8d37872: 2023-02-19 12:58:54,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119. 2023-02-19 12:58:54,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 92751fca21674e6b7ab33630e2b6a119: 2023-02-19 12:58:54,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/.tmp/cf/74cb70d047e24fa9bf07cd1d3cd5a1ee 2023-02-19 12:58:54,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:54,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:54,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9afa96a011d8f82b0bebbd3bac2f1fce, disabling compactions & flushes 2023-02-19 12:58:54,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:54,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:54,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. after waiting 0 ms 2023-02-19 12:58:54,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:54,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9afa96a011d8f82b0bebbd3bac2f1fce 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-19 12:58:54,698 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=92751fca21674e6b7ab33630e2b6a119, regionState=CLOSED 2023-02-19 12:58:54,699 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534698"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811534698"}]},"ts":"1676811534698"} 2023-02-19 12:58:54,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:54,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:54,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0056208a08192d81a1d3f47ace8488e5, disabling compactions & flushes 2023-02-19 12:58:54,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:54,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:54,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. after waiting 0 ms 2023-02-19 12:58:54,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:54,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 0056208a08192d81a1d3f47ace8488e5 1/1 column families, dataSize=22.44 KB heapSize=48.48 KB 2023-02-19 12:58:54,699 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=9c525e8dcdbb974e61fda246b8d37872, regionState=CLOSED 2023-02-19 12:58:54,699 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811534699"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811534699"}]},"ts":"1676811534699"} 2023-02-19 12:58:54,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/.tmp/cf/74cb70d047e24fa9bf07cd1d3cd5a1ee as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/74cb70d047e24fa9bf07cd1d3cd5a1ee 2023-02-19 12:58:54,702 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=273 2023-02-19 12:58:54,702 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=273, state=SUCCESS; CloseRegionProcedure 92751fca21674e6b7ab33630e2b6a119, server=jenkins-hbase4.apache.org,45077,1676811501769 in 197 msec 2023-02-19 12:58:54,703 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=271 2023-02-19 12:58:54,703 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=271, state=SUCCESS; CloseRegionProcedure 9c525e8dcdbb974e61fda246b8d37872, server=jenkins-hbase4.apache.org,41307,1676811501696 in 200 msec 2023-02-19 12:58:54,705 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=92751fca21674e6b7ab33630e2b6a119, UNASSIGN in 203 msec 2023-02-19 12:58:54,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9c525e8dcdbb974e61fda246b8d37872, UNASSIGN in 204 msec 2023-02-19 12:58:54,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/74cb70d047e24fa9bf07cd1d3cd5a1ee, entries=39, sequenceid=15, filesize=11.6 K 2023-02-19 12:58:54,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for c6d304ba41abeb8e6ae55cbd4c42f7c1 in 51ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:54,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:54,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1. 2023-02-19 12:58:54,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c6d304ba41abeb8e6ae55cbd4c42f7c1: 2023-02-19 12:58:54,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:54,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:54,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4aca2f2bf5da4c990f8539f674ac2e3e, disabling compactions & flushes 2023-02-19 12:58:54,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:54,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:54,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. after waiting 0 ms 2023-02-19 12:58:54,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:54,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 4aca2f2bf5da4c990f8539f674ac2e3e 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-19 12:58:54,730 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=c6d304ba41abeb8e6ae55cbd4c42f7c1, regionState=CLOSED 2023-02-19 12:58:54,730 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534730"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811534730"}]},"ts":"1676811534730"} 2023-02-19 12:58:54,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021985d88119745747ec8331ad8e63f9ed1a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021985d88119745747ec8331ad8e63f9ed1a 2023-02-19 12:58:54,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=272 2023-02-19 12:58:54,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=272, state=SUCCESS; CloseRegionProcedure c6d304ba41abeb8e6ae55cbd4c42f7c1, server=jenkins-hbase4.apache.org,39199,1676811501737 in 229 msec 2023-02-19 12:58:54,735 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=c6d304ba41abeb8e6ae55cbd4c42f7c1, UNASSIGN in 235 msec 2023-02-19 12:58:54,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-19 12:58:54,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302197a5cfaea6baa42899af8b0b6083aad9b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d5202302197a5cfaea6baa42899af8b0b6083aad9b 2023-02-19 12:58:54,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/.tmp/cf/4f3cd9ff77d34ad49a744c61ef688d76 2023-02-19 12:58:54,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/.tmp/cf/4f3cd9ff77d34ad49a744c61ef688d76 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/4f3cd9ff77d34ad49a744c61ef688d76 2023-02-19 12:58:54,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/.tmp/cf/dd5355ea20a84019a7126590225264df 2023-02-19 12:58:54,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/4f3cd9ff77d34ad49a744c61ef688d76, entries=24, sequenceid=15, filesize=9.0 K 2023-02-19 12:58:54,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 9afa96a011d8f82b0bebbd3bac2f1fce in 76ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:54,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/.tmp/cf/dd5355ea20a84019a7126590225264df as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/dd5355ea20a84019a7126590225264df 2023-02-19 12:58:54,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:54,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce. 2023-02-19 12:58:54,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9afa96a011d8f82b0bebbd3bac2f1fce: 2023-02-19 12:58:54,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/dd5355ea20a84019a7126590225264df, entries=343, sequenceid=15, filesize=64.0 K 2023-02-19 12:58:54,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.44 KB/22977, heapSize ~48.47 KB/49632, currentSize=0 B/0 for 0056208a08192d81a1d3f47ace8488e5 in 80ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:54,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:54,779 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=9afa96a011d8f82b0bebbd3bac2f1fce, regionState=CLOSED 2023-02-19 12:58:54,779 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811534779"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811534779"}]},"ts":"1676811534779"} 2023-02-19 12:58:54,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:54,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=274 2023-02-19 12:58:54,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=274, state=SUCCESS; CloseRegionProcedure 9afa96a011d8f82b0bebbd3bac2f1fce, server=jenkins-hbase4.apache.org,45077,1676811501769 in 279 msec 2023-02-19 12:58:54,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5. 2023-02-19 12:58:54,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0056208a08192d81a1d3f47ace8488e5: 2023-02-19 12:58:54,784 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=9afa96a011d8f82b0bebbd3bac2f1fce, UNASSIGN in 283 msec 2023-02-19 12:58:54,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:54,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=0056208a08192d81a1d3f47ace8488e5, regionState=CLOSED 2023-02-19 12:58:54,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676811534784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811534784"}]},"ts":"1676811534784"} 2023-02-19 12:58:54,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=276 2023-02-19 12:58:54,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=276, state=SUCCESS; CloseRegionProcedure 0056208a08192d81a1d3f47ace8488e5, server=jenkins-hbase4.apache.org,41307,1676811501696 in 283 msec 2023-02-19 12:58:54,787 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=0056208a08192d81a1d3f47ace8488e5, UNASSIGN in 288 msec 2023-02-19 12:58:55,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021970f98ad5389c4940a8d2f4adb4ae57b9 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021970f98ad5389c4940a8d2f4adb4ae57b9 2023-02-19 12:58:55,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/.tmp/cf/c17533aa25f94a38a7c6d949e06bfd60 2023-02-19 12:58:55,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/.tmp/cf/c17533aa25f94a38a7c6d949e06bfd60 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/c17533aa25f94a38a7c6d949e06bfd60 2023-02-19 12:58:55,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/c17533aa25f94a38a7c6d949e06bfd60, entries=28, sequenceid=15, filesize=9.7 K 2023-02-19 12:58:55,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 4aca2f2bf5da4c990f8539f674ac2e3e in 453ms, sequenceid=15, compaction requested=false 2023-02-19 12:58:55,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-19 12:58:55,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e. 2023-02-19 12:58:55,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4aca2f2bf5da4c990f8539f674ac2e3e: 2023-02-19 12:58:55,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:55,188 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=4aca2f2bf5da4c990f8539f674ac2e3e, regionState=CLOSED 2023-02-19 12:58:55,189 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676811535188"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676811535188"}]},"ts":"1676811535188"} 2023-02-19 12:58:55,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=275 2023-02-19 12:58:55,191 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=275, state=SUCCESS; CloseRegionProcedure 4aca2f2bf5da4c990f8539f674ac2e3e, server=jenkins-hbase4.apache.org,39199,1676811501737 in 684 msec 2023-02-19 12:58:55,191 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=275, resume processing ppid=270 2023-02-19 12:58:55,191 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676811528891, region=4aca2f2bf5da4c990f8539f674ac2e3e, UNASSIGN in 691 msec 2023-02-19 12:58:55,192 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676811535192"}]},"ts":"1676811535192"} 2023-02-19 12:58:55,193 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676811528891, state=DISABLED in hbase:meta 2023-02-19 12:58:55,194 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676811528891 to state=DISABLED 2023-02-19 12:58:55,195 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676811528891 in 702 msec 2023-02-19 12:58:55,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-19 12:58:55,248 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 270 completed 2023-02-19 12:58:55,248 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,250 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-19 12:58:55,257 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:55,258 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:55,258 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:55,258 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:55,258 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:55,257 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:55,260 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits] 2023-02-19 12:58:55,260 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits] 2023-02-19 12:58:55,260 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits] 2023-02-19 12:58:55,261 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits] 2023-02-19 12:58:55,261 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits] 2023-02-19 12:58:55,261 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf, FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits] 2023-02-19 12:58:55,265 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/0be92ab043194f2383f6e3a9c100e97e to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/0be92ab043194f2383f6e3a9c100e97e 2023-02-19 12:58:55,265 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/01fdd656a5fa4385bf5633fe0a572a69 2023-02-19 12:58:55,266 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/4f3cd9ff77d34ad49a744c61ef688d76 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/4f3cd9ff77d34ad49a744c61ef688d76 2023-02-19 12:58:55,266 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/6906358dd2884f3ca1e06713719c580d 2023-02-19 12:58:55,266 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/74cb70d047e24fa9bf07cd1d3cd5a1ee to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/74cb70d047e24fa9bf07cd1d3cd5a1ee 2023-02-19 12:58:55,266 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/dd5355ea20a84019a7126590225264df to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/cf/dd5355ea20a84019a7126590225264df 2023-02-19 12:58:55,267 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/cf/18895ddcab074cc6a6095930dcf5d793 2023-02-19 12:58:55,267 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/cf/cba708de7bc749579ba8d4268b7bb26d 2023-02-19 12:58:55,267 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/374a57461bb74402a4acf0c0242ddc36 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/cf/374a57461bb74402a4acf0c0242ddc36 2023-02-19 12:58:55,267 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/c17533aa25f94a38a7c6d949e06bfd60 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/cf/c17533aa25f94a38a7c6d949e06bfd60 2023-02-19 12:58:55,268 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/cf/f7abb3c68c6c4328affbf68d38caa446 2023-02-19 12:58:55,273 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5/recovered.edits/18.seqid 2023-02-19 12:58:55,273 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce/recovered.edits/18.seqid 2023-02-19 12:58:55,274 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/0056208a08192d81a1d3f47ace8488e5 2023-02-19 12:58:55,274 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872/recovered.edits/18.seqid 2023-02-19 12:58:55,274 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9afa96a011d8f82b0bebbd3bac2f1fce 2023-02-19 12:58:55,274 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119/recovered.edits/18.seqid 2023-02-19 12:58:55,274 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e/recovered.edits/18.seqid 2023-02-19 12:58:55,275 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits/18.seqid to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1/recovered.edits/18.seqid 2023-02-19 12:58:55,275 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/9c525e8dcdbb974e61fda246b8d37872 2023-02-19 12:58:55,275 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/92751fca21674e6b7ab33630e2b6a119 2023-02-19 12:58:55,275 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/4aca2f2bf5da4c990f8539f674ac2e3e 2023-02-19 12:58:55,275 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/.tmp/data/default/testCorruptedSnapshot-1676811528891/c6d304ba41abeb8e6ae55cbd4c42f7c1 2023-02-19 12:58:55,275 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676811528891 regions 2023-02-19 12:58:55,275 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc 2023-02-19 12:58:55,276 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf] 2023-02-19 12:58:55,279 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021935bdd2b1ff89445fb3532b37fe28a778 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021935bdd2b1ff89445fb3532b37fe28a778 2023-02-19 12:58:55,280 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021970f98ad5389c4940a8d2f4adb4ae57b9 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/a87ff679a2f3e71d9181a67b7542122c2023021970f98ad5389c4940a8d2f4adb4ae57b9 2023-02-19 12:58:55,280 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219731fc428f5da482391cbbca8974cad47 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219731fc428f5da482391cbbca8974cad47 2023-02-19 12:58:55,281 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219c98e9edb9ed1442a957e4e318b8431ab to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c4ca4238a0b923820dcc509a6f75849b20230219c98e9edb9ed1442a957e4e318b8431ab 2023-02-19 12:58:55,282 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219266d8e9613bd43069140deb8608a4f00 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219266d8e9613bd43069140deb8608a4f00 2023-02-19 12:58:55,283 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219805b5aff2d5e4304b829a115b84beb78 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/c81e728d9d4c2f636f067f89cc14862c20230219805b5aff2d5e4304b829a115b84beb78 2023-02-19 12:58:55,284 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e202302196d3aa93b6dc54d0db23f5e67224fbec9 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e202302196d3aa93b6dc54d0db23f5e67224fbec9 2023-02-19 12:58:55,285 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e20230219712c20746b9441edaab22fb6c533ddb6 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/d41d8cd98f00b204e9800998ecf8427e20230219712c20746b9441edaab22fb6c533ddb6 2023-02-19 12:58:55,286 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d5202302197a5cfaea6baa42899af8b0b6083aad9b to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d5202302197a5cfaea6baa42899af8b0b6083aad9b 2023-02-19 12:58:55,287 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d520230219c842bb5938004f4590c55a77b9cf255d to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/e4da3b7fbbce2345d7772b0674a318d520230219c842bb5938004f4590c55a77b9cf255d 2023-02-19 12:58:55,288 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021985d88119745747ec8331ad8e63f9ed1a to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021985d88119745747ec8331ad8e63f9ed1a 2023-02-19 12:58:55,288 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219e08f0f0bb3aa44b7ad373721f2beae45 to hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/archive/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230219e08f0f0bb3aa44b7ad373721f2beae45 2023-02-19 12:58:55,289 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/mobdir/data/default/testCorruptedSnapshot-1676811528891/b3e4dc0756b78a798afcb9ac4c9dfffc 2023-02-19 12:58:55,290 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,293 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676811528891 from hbase:meta 2023-02-19 12:58:55,294 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676811528891' descriptor. 2023-02-19 12:58:55,295 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676811528891' from region states. 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811535296"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811535296"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811535296"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811535296"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811535296"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,296 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676811535296"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,298 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-19 12:58:55,298 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9c525e8dcdbb974e61fda246b8d37872, NAME => 'testCorruptedSnapshot-1676811528891,,1676811528893.9c525e8dcdbb974e61fda246b8d37872.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => c6d304ba41abeb8e6ae55cbd4c42f7c1, NAME => 'testCorruptedSnapshot-1676811528891,1,1676811528893.c6d304ba41abeb8e6ae55cbd4c42f7c1.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 92751fca21674e6b7ab33630e2b6a119, NAME => 'testCorruptedSnapshot-1676811528891,2,1676811528893.92751fca21674e6b7ab33630e2b6a119.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 9afa96a011d8f82b0bebbd3bac2f1fce, NAME => 'testCorruptedSnapshot-1676811528891,3,1676811528893.9afa96a011d8f82b0bebbd3bac2f1fce.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4aca2f2bf5da4c990f8539f674ac2e3e, NAME => 'testCorruptedSnapshot-1676811528891,4,1676811528893.4aca2f2bf5da4c990f8539f674ac2e3e.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 0056208a08192d81a1d3f47ace8488e5, NAME => 'testCorruptedSnapshot-1676811528891,5,1676811528893.0056208a08192d81a1d3f47ace8488e5.', STARTKEY => '5', ENDKEY => ''}] 2023-02-19 12:58:55,298 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676811528891' as deleted. 2023-02-19 12:58:55,298 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676811528891","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676811535298"}]},"ts":"9223372036854775807"} 2023-02-19 12:58:55,299 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676811528891 state from META 2023-02-19 12:58:55,302 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 2023-02-19 12:58:55,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676811528891 in 54 msec 2023-02-19 12:58:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-19 12:58:55,502 INFO [Listener at localhost/43399] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676811528891, procId: 283 completed 2023-02-19 12:58:55,506 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676811528891" 2023-02-19 12:58:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676811528891 2023-02-19 12:58:55,509 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676811528891" 2023-02-19 12:58:55,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34259] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676811528891 2023-02-19 12:58:55,528 INFO [Listener at localhost/43399] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=420 (was 425), OpenFileDescriptor=805 (was 805), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=92 (was 109), ProcessCount=170 (was 170), AvailableMemoryMB=9355 (was 9380) 2023-02-19 12:58:55,529 INFO [Listener at localhost/43399] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-19 12:58:55,529 INFO [Listener at localhost/43399] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-19 12:58:55,530 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x297db6e1 to 127.0.0.1:60853 2023-02-19 12:58:55,530 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,530 DEBUG [Listener at localhost/43399] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-19 12:58:55,530 DEBUG [Listener at localhost/43399] util.JVMClusterUtil(257): Found active master hash=780634610, stopped=false 2023-02-19 12:58:55,530 INFO [Listener at localhost/43399] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:55,532 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:55,532 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:55,532 INFO [Listener at localhost/43399] procedure2.ProcedureExecutor(629): Stopping 2023-02-19 12:58:55,532 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:55,532 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-19 12:58:55,533 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:55,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:55,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:55,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:55,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-19 12:58:55,533 DEBUG [Listener at localhost/43399] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2961df43 to 127.0.0.1:60853 2023-02-19 12:58:55,533 DEBUG [Listener at localhost/43399] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,534 INFO [Listener at localhost/43399] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41307,1676811501696' ***** 2023-02-19 12:58:55,534 INFO [Listener at localhost/43399] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-19 12:58:55,534 INFO [Listener at localhost/43399] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39199,1676811501737' ***** 2023-02-19 12:58:55,534 INFO [Listener at localhost/43399] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-19 12:58:55,534 INFO [Listener at localhost/43399] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45077,1676811501769' ***** 2023-02-19 12:58:55,534 INFO [Listener at localhost/43399] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-19 12:58:55,534 INFO [RS:1;jenkins-hbase4:39199] regionserver.HeapMemoryManager(220): Stopping 2023-02-19 12:58:55,534 INFO [RS:2;jenkins-hbase4:45077] regionserver.HeapMemoryManager(220): Stopping 2023-02-19 12:58:55,535 INFO [RS:2;jenkins-hbase4:45077] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-19 12:58:55,534 INFO [RS:0;jenkins-hbase4:41307] regionserver.HeapMemoryManager(220): Stopping 2023-02-19 12:58:55,535 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-19 12:58:55,535 INFO [RS:0;jenkins-hbase4:41307] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-19 12:58:55,535 INFO [RS:2;jenkins-hbase4:45077] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-19 12:58:55,535 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-19 12:58:55,535 INFO [RS:1;jenkins-hbase4:39199] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-19 12:58:55,535 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:55,535 INFO [RS:1;jenkins-hbase4:39199] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-19 12:58:55,535 INFO [RS:0;jenkins-hbase4:41307] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-19 12:58:55,535 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-19 12:58:55,535 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:55,535 DEBUG [RS:2;jenkins-hbase4:45077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32fd08ad to 127.0.0.1:60853 2023-02-19 12:58:55,535 DEBUG [RS:0;jenkins-hbase4:41307] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cfee55f to 127.0.0.1:60853 2023-02-19 12:58:55,535 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(3304): Received CLOSE for 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:55,536 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:55,536 DEBUG [RS:1;jenkins-hbase4:39199] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x280df3d3 to 127.0.0.1:60853 2023-02-19 12:58:55,536 DEBUG [RS:1;jenkins-hbase4:39199] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,536 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-19 12:58:55,536 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1479): Online Regions={27c253a2f5b636a8349cb7b2efa921a1=hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1.} 2023-02-19 12:58:55,535 DEBUG [RS:0;jenkins-hbase4:41307] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,535 DEBUG [RS:2;jenkins-hbase4:45077] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,536 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41307,1676811501696; all regions closed. 2023-02-19 12:58:55,536 INFO [RS:2;jenkins-hbase4:45077] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-19 12:58:55,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 27c253a2f5b636a8349cb7b2efa921a1, disabling compactions & flushes 2023-02-19 12:58:55,536 INFO [RS:2;jenkins-hbase4:45077] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-19 12:58:55,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:55,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:55,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. after waiting 0 ms 2023-02-19 12:58:55,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:55,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 27c253a2f5b636a8349cb7b2efa921a1 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-19 12:58:55,536 INFO [RS:2;jenkins-hbase4:45077] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-19 12:58:55,537 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-19 12:58:55,537 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-19 12:58:55,538 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-19 12:58:55,538 DEBUG [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1505): Waiting on 27c253a2f5b636a8349cb7b2efa921a1 2023-02-19 12:58:55,538 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-19 12:58:55,538 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-19 12:58:55,538 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-19 12:58:55,538 DEBUG [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-19 12:58:55,538 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-19 12:58:55,538 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-19 12:58:55,538 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-19 12:58:55,545 DEBUG [RS:0;jenkins-hbase4:41307] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs 2023-02-19 12:58:55,545 INFO [RS:0;jenkins-hbase4:41307] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41307%2C1676811501696:(num 1676811503177) 2023-02-19 12:58:55,545 DEBUG [RS:0;jenkins-hbase4:41307] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,545 INFO [RS:0;jenkins-hbase4:41307] regionserver.LeaseManager(133): Closed leases 2023-02-19 12:58:55,546 INFO [RS:0;jenkins-hbase4:41307] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-19 12:58:55,546 INFO [RS:0;jenkins-hbase4:41307] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-19 12:58:55,546 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-19 12:58:55,546 INFO [RS:0;jenkins-hbase4:41307] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-19 12:58:55,546 INFO [RS:0;jenkins-hbase4:41307] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-19 12:58:55,548 INFO [RS:0;jenkins-hbase4:41307] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41307 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41307,1676811501696 2023-02-19 12:58:55,560 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:55,561 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41307,1676811501696] 2023-02-19 12:58:55,561 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41307,1676811501696; numProcessing=1 2023-02-19 12:58:55,562 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41307,1676811501696 already deleted, retry=false 2023-02-19 12:58:55,562 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41307,1676811501696 expired; onlineServers=2 2023-02-19 12:58:55,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/.tmp/info/af7fa941bd904712a1fc3b22dbfd9aaa 2023-02-19 12:58:55,566 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/.tmp/info/f6125dfd6b3e49fa878d2138b611e480 2023-02-19 12:58:55,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/.tmp/info/af7fa941bd904712a1fc3b22dbfd9aaa as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/info/af7fa941bd904712a1fc3b22dbfd9aaa 2023-02-19 12:58:55,570 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f6125dfd6b3e49fa878d2138b611e480 2023-02-19 12:58:55,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/info/af7fa941bd904712a1fc3b22dbfd9aaa, entries=2, sequenceid=6, filesize=4.8 K 2023-02-19 12:58:55,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 27c253a2f5b636a8349cb7b2efa921a1 in 36ms, sequenceid=6, compaction requested=false 2023-02-19 12:58:55,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/namespace/27c253a2f5b636a8349cb7b2efa921a1/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-19 12:58:55,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:55,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 27c253a2f5b636a8349cb7b2efa921a1: 2023-02-19 12:58:55,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676811503626.27c253a2f5b636a8349cb7b2efa921a1. 2023-02-19 12:58:55,586 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/.tmp/rep_barrier/4dde91d515bc4aaa892084a70cacbdad 2023-02-19 12:58:55,589 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4dde91d515bc4aaa892084a70cacbdad 2023-02-19 12:58:55,596 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-19 12:58:55,596 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-19 12:58:55,596 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-19 12:58:55,600 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/.tmp/table/44754bde872249abac3b1d1164bda5cc 2023-02-19 12:58:55,603 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 44754bde872249abac3b1d1164bda5cc 2023-02-19 12:58:55,603 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/.tmp/info/f6125dfd6b3e49fa878d2138b611e480 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/info/f6125dfd6b3e49fa878d2138b611e480 2023-02-19 12:58:55,606 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f6125dfd6b3e49fa878d2138b611e480 2023-02-19 12:58:55,606 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/info/f6125dfd6b3e49fa878d2138b611e480, entries=64, sequenceid=307, filesize=11.8 K 2023-02-19 12:58:55,607 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/.tmp/rep_barrier/4dde91d515bc4aaa892084a70cacbdad as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/rep_barrier/4dde91d515bc4aaa892084a70cacbdad 2023-02-19 12:58:55,609 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4dde91d515bc4aaa892084a70cacbdad 2023-02-19 12:58:55,609 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/rep_barrier/4dde91d515bc4aaa892084a70cacbdad, entries=18, sequenceid=307, filesize=7.0 K 2023-02-19 12:58:55,610 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/.tmp/table/44754bde872249abac3b1d1164bda5cc as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/table/44754bde872249abac3b1d1164bda5cc 2023-02-19 12:58:55,613 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 44754bde872249abac3b1d1164bda5cc 2023-02-19 12:58:55,613 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/table/44754bde872249abac3b1d1164bda5cc, entries=22, sequenceid=307, filesize=7.0 K 2023-02-19 12:58:55,613 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 75ms, sequenceid=307, compaction requested=false 2023-02-19 12:58:55,617 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-19 12:58:55,618 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-19 12:58:55,619 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-19 12:58:55,619 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-19 12:58:55,619 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-19 12:58:55,733 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:55,733 INFO [RS:0;jenkins-hbase4:41307] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41307,1676811501696; zookeeper connection closed. 2023-02-19 12:58:55,734 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5f2bd4ca rejected from java.util.concurrent.ThreadPoolExecutor@5b144bf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:55,734 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@22367bfa] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@22367bfa 2023-02-19 12:58:55,734 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:41307-0x1006ca8e2820001, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:55,734 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ece3dac rejected from java.util.concurrent.ThreadPoolExecutor@5b144bf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:55,738 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39199,1676811501737; all regions closed. 2023-02-19 12:58:55,738 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45077,1676811501769; all regions closed. 2023-02-19 12:58:55,745 DEBUG [RS:1;jenkins-hbase4:39199] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs 2023-02-19 12:58:55,745 INFO [RS:1;jenkins-hbase4:39199] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39199%2C1676811501737:(num 1676811503177) 2023-02-19 12:58:55,745 DEBUG [RS:1;jenkins-hbase4:39199] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,745 INFO [RS:1;jenkins-hbase4:39199] regionserver.LeaseManager(133): Closed leases 2023-02-19 12:58:55,746 INFO [RS:1;jenkins-hbase4:39199] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-19 12:58:55,746 INFO [RS:1;jenkins-hbase4:39199] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-19 12:58:55,746 INFO [RS:1;jenkins-hbase4:39199] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-19 12:58:55,746 INFO [RS:1;jenkins-hbase4:39199] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-19 12:58:55,746 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-19 12:58:55,746 DEBUG [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs 2023-02-19 12:58:55,747 INFO [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45077%2C1676811501769.meta:.meta(num 1676811503363) 2023-02-19 12:58:55,747 INFO [RS:1;jenkins-hbase4:39199] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39199 2023-02-19 12:58:55,751 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:55,751 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39199,1676811501737 2023-02-19 12:58:55,751 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:55,752 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39199,1676811501737] 2023-02-19 12:58:55,752 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39199,1676811501737; numProcessing=2 2023-02-19 12:58:55,753 DEBUG [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/oldWALs 2023-02-19 12:58:55,753 INFO [RS:2;jenkins-hbase4:45077] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45077%2C1676811501769:(num 1676811503177) 2023-02-19 12:58:55,753 DEBUG [RS:2;jenkins-hbase4:45077] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,753 INFO [RS:2;jenkins-hbase4:45077] regionserver.LeaseManager(133): Closed leases 2023-02-19 12:58:55,753 INFO [RS:2;jenkins-hbase4:45077] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-19 12:58:55,753 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39199,1676811501737 already deleted, retry=false 2023-02-19 12:58:55,753 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-19 12:58:55,753 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39199,1676811501737 expired; onlineServers=1 2023-02-19 12:58:55,754 INFO [RS:2;jenkins-hbase4:45077] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45077 2023-02-19 12:58:55,756 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45077,1676811501769 2023-02-19 12:58:55,756 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-19 12:58:55,756 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45077,1676811501769] 2023-02-19 12:58:55,756 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45077,1676811501769; numProcessing=3 2023-02-19 12:58:55,758 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45077,1676811501769 already deleted, retry=false 2023-02-19 12:58:55,758 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,45077,1676811501769 expired; onlineServers=0 2023-02-19 12:58:55,758 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34259,1676811500546' ***** 2023-02-19 12:58:55,758 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-19 12:58:55,759 DEBUG [M:0;jenkins-hbase4:34259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fba83c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-19 12:58:55,759 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34259,1676811500546 2023-02-19 12:58:55,759 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34259,1676811500546; all regions closed. 2023-02-19 12:58:55,759 DEBUG [M:0;jenkins-hbase4:34259] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-19 12:58:55,759 DEBUG [M:0;jenkins-hbase4:34259] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-19 12:58:55,759 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-19 12:58:55,759 DEBUG [M:0;jenkins-hbase4:34259] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-19 12:58:55,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676811502832] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676811502832,5,FailOnTimeoutGroup] 2023-02-19 12:58:55,759 INFO [M:0;jenkins-hbase4:34259] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-19 12:58:55,759 INFO [M:0;jenkins-hbase4:34259] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-19 12:58:55,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676811502832] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676811502832,5,FailOnTimeoutGroup] 2023-02-19 12:58:55,760 INFO [M:0;jenkins-hbase4:34259] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-19 12:58:55,760 DEBUG [M:0;jenkins-hbase4:34259] master.HMaster(1512): Stopping service threads 2023-02-19 12:58:55,760 INFO [M:0;jenkins-hbase4:34259] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-19 12:58:55,760 ERROR [M:0;jenkins-hbase4:34259] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-19 12:58:55,761 INFO [M:0;jenkins-hbase4:34259] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-19 12:58:55,761 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-19 12:58:55,761 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-19 12:58:55,761 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-19 12:58:55,762 DEBUG [M:0;jenkins-hbase4:34259] zookeeper.ZKUtil(398): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-19 12:58:55,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-19 12:58:55,762 WARN [M:0;jenkins-hbase4:34259] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-19 12:58:55,762 INFO [M:0;jenkins-hbase4:34259] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-19 12:58:55,762 INFO [M:0;jenkins-hbase4:34259] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-19 12:58:55,762 DEBUG [M:0;jenkins-hbase4:34259] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-19 12:58:55,762 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:55,762 DEBUG [M:0;jenkins-hbase4:34259] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:55,762 DEBUG [M:0;jenkins-hbase4:34259] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-19 12:58:55,762 DEBUG [M:0;jenkins-hbase4:34259] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:55,762 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.48 KB heapSize=1.01 MB 2023-02-19 12:58:55,777 INFO [M:0;jenkins-hbase4:34259] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.48 KB at sequenceid=2109 (bloomFilter=true), to=hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/252e63dc739d482caa209b73d4dcdbd3 2023-02-19 12:58:55,780 INFO [M:0;jenkins-hbase4:34259] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 252e63dc739d482caa209b73d4dcdbd3 2023-02-19 12:58:55,781 DEBUG [M:0;jenkins-hbase4:34259] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/252e63dc739d482caa209b73d4dcdbd3 as hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/252e63dc739d482caa209b73d4dcdbd3 2023-02-19 12:58:55,784 INFO [M:0;jenkins-hbase4:34259] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 252e63dc739d482caa209b73d4dcdbd3 2023-02-19 12:58:55,784 INFO [M:0;jenkins-hbase4:34259] regionserver.HStore(1080): Added hdfs://localhost:36945/user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/252e63dc739d482caa209b73d4dcdbd3, entries=282, sequenceid=2109, filesize=30.9 K 2023-02-19 12:58:55,785 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegion(2947): Finished flush of dataSize ~845.48 KB/865772, heapSize ~1.01 MB/1061872, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=2109, compaction requested=false 2023-02-19 12:58:55,785 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-19 12:58:55,785 DEBUG [M:0;jenkins-hbase4:34259] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-19 12:58:55,790 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/ce3c8687-a229-27f9-cd85-dbc3c5b5b461/MasterData/WALs/jenkins-hbase4.apache.org,34259,1676811500546/jenkins-hbase4.apache.org%2C34259%2C1676811500546.1676811502195 not finished, retry = 0 2023-02-19 12:58:55,891 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-19 12:58:55,891 INFO [M:0;jenkins-hbase4:34259] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-19 12:58:55,892 INFO [M:0;jenkins-hbase4:34259] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34259 2023-02-19 12:58:55,893 DEBUG [M:0;jenkins-hbase4:34259] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34259,1676811500546 already deleted, retry=false 2023-02-19 12:58:56,235 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:56,235 INFO [M:0;jenkins-hbase4:34259] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34259,1676811500546; zookeeper connection closed. 2023-02-19 12:58:56,235 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5ac865ce rejected from java.util.concurrent.ThreadPoolExecutor@2c4d1e16[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:56,235 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): master:34259-0x1006ca8e2820000, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:56,235 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a4f439d rejected from java.util.concurrent.ThreadPoolExecutor@2c4d1e16[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:56,335 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:56,335 INFO [RS:2;jenkins-hbase4:45077] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45077,1676811501769; zookeeper connection closed. 2023-02-19 12:58:56,335 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@13091f4b rejected from java.util.concurrent.ThreadPoolExecutor@173689f2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:56,335 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:45077-0x1006ca8e2820003, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:56,335 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38842746 rejected from java.util.concurrent.ThreadPoolExecutor@173689f2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:56,335 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@13c20fb2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@13c20fb2 2023-02-19 12:58:56,435 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:56,435 INFO [RS:1;jenkins-hbase4:39199] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39199,1676811501737; zookeeper connection closed. 2023-02-19 12:58:56,435 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1dfc49ea rejected from java.util.concurrent.ThreadPoolExecutor@19c3ce8d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:56,435 DEBUG [Listener at localhost/43399-EventThread] zookeeper.ZKWatcher(600): regionserver:39199-0x1006ca8e2820002, quorum=127.0.0.1:60853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-19 12:58:56,435 ERROR [Listener at localhost/43399-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@129b11f9 rejected from java.util.concurrent.ThreadPoolExecutor@19c3ce8d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-19 12:58:56,435 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@71e5ca5c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@71e5ca5c 2023-02-19 12:58:56,436 INFO [Listener at localhost/43399] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-19 12:58:56,436 WARN [Listener at localhost/43399] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-19 12:58:56,438 INFO [Listener at localhost/43399] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-19 12:58:56,543 WARN [BP-1712938287-172.31.14.131-1676811497500 heartbeating to localhost/127.0.0.1:36945] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-19 12:58:56,543 WARN [BP-1712938287-172.31.14.131-1676811497500 heartbeating to localhost/127.0.0.1:36945] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1712938287-172.31.14.131-1676811497500 (Datanode Uuid 78813b76-a43f-4947-becb-21ca672de048) service to localhost/127.0.0.1:36945 2023-02-19 12:58:56,545 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data5/current/BP-1712938287-172.31.14.131-1676811497500] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-19 12:58:56,545 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data6/current/BP-1712938287-172.31.14.131-1676811497500] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-19 12:58:56,546 WARN [Listener at localhost/43399] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-19 12:58:56,548 INFO [Listener at localhost/43399] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-19 12:58:56,653 WARN [BP-1712938287-172.31.14.131-1676811497500 heartbeating to localhost/127.0.0.1:36945] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-19 12:58:56,653 WARN [BP-1712938287-172.31.14.131-1676811497500 heartbeating to localhost/127.0.0.1:36945] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1712938287-172.31.14.131-1676811497500 (Datanode Uuid f708cb9d-f08b-4be3-ba60-a65026033f70) service to localhost/127.0.0.1:36945 2023-02-19 12:58:56,654 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data3/current/BP-1712938287-172.31.14.131-1676811497500] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-19 12:58:56,654 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data4/current/BP-1712938287-172.31.14.131-1676811497500] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-19 12:58:56,655 WARN [Listener at localhost/43399] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-19 12:58:56,657 INFO [Listener at localhost/43399] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-19 12:58:56,762 WARN [BP-1712938287-172.31.14.131-1676811497500 heartbeating to localhost/127.0.0.1:36945] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-19 12:58:56,762 WARN [BP-1712938287-172.31.14.131-1676811497500 heartbeating to localhost/127.0.0.1:36945] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1712938287-172.31.14.131-1676811497500 (Datanode Uuid 71e52b31-5255-4bd9-9704-0a46b7647275) service to localhost/127.0.0.1:36945 2023-02-19 12:58:56,762 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data1/current/BP-1712938287-172.31.14.131-1676811497500] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-19 12:58:56,763 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e96dc8c-f289-6f92-b3e6-27e1a6e7629d/cluster_6d9f332b-320e-1e1c-3542-29add5fe5ae5/dfs/data/data2/current/BP-1712938287-172.31.14.131-1676811497500] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-19 12:58:56,790 INFO [Listener at localhost/43399] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-19 12:58:56,916 INFO [Listener at localhost/43399] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-19 12:58:56,995 INFO [Listener at localhost/43399] hbase.HBaseTestingUtility(1293): Minicluster is down